Let's make Godzilla's image recognition model preprocessing, learning and deployment feel good

This article is the 20th day of "Chura Data Advent Calendar".

This is a half-finished article even though it is late. I will complete it tomorrow.

Overview

Recently, I'm interested in MLOps, and I'm hunting for information and reading books.

It is a Godzilla image classification API that was shown at the Chura Data Anniversary Festival, but when I fine-tuned the VGG16 model and learned using the images collected by Google image search appropriately, it can be classified with a loose feeling. became.

Such a guy

image.png

However, when I showed it, black people are classified as Godzilla, Gamera is classified as Godzilla, and it seems that learning and data are still insufficient.

① Collect data → ② Label → ③ Data placement → ④ Hit the Jupyter notebook for learning from above → ⑤ Check the result

However, I do the cycle manually, but it is troublesome to execute it many times. That's why I wanted to make it a workflow.

Regarding (1), it has already been implemented in code, and for (2), it seems difficult if people do not do it for the first time, so I will do it manually together with (3). This time, I thought it would be great if only ④ and ⑤ could be made into a workflow.

Also, if it is about this scale, it seems that sklearn's pipeline can be used, but I was looking for a good framework because it is troublesome to implement the data download process again.

Then I found this article.

MLOps2020 to start small and grow big

When I check the article above, it says that Kedro + Mlflow is good, so I thought I should try it for the first time.

That's why I prepared it, so I will write about the implementation method.

I wanted to prepare it quickly, but I couldn't.

environment

Local environment

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.7
BuildVersion:	19H15

$ docker -v
Docker version 19.03.13, build 4484c46d9d

$ docker-compose -v
docker-compose version 1.27.4, build 40524192

Preparation

AWS

Play in Oregon.

Preparing IAM users

--Note that AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY will be used later. --S3 Please have write / read permission

S3

--Prepare a bucket for the time being --This time, I prepared it with godzilla-hogehoge-1234 (I will erase it anyway)

EC2

--Use p2.xlarge (tears) ――Spot instances are cheap! !! !!

Dockerfile ready

I will do it with docker

.env

Add the AWS information you wrote down earlier.


#Postgres information
POSTGRES_USER=user
POSTGRES_PASSWORD=password

#Prepare the created s3 bucket name
S3_BUCKET=s3://godzilla-hogehoge-1234/

#AWS information
AWS_ACCESS_KEY_ID=hogehoge
AWS_SECRET_ACCESS_KEY=fugafuga
AWS_DEFAULT_REGION=us-west-2
AWS_DEFAULT_OUTPUT=json

Dockerfile for kedro

Create with Dockerfile_kedro.


FROM hoto17296/anaconda3-ja

RUN pip install kedro==0.17.0 \
    fsspec==0.6.3 \
    s3fs==0.4.0 \
    botocore==1.19.36 \
    mlflow==1.12.1 \
    tensorflow-gpu

Dockerfile for mlflow server

Create with Dockerfile_mlflow.

By the way, I respected the code etc. from ↓ Pakuri .

https://github.com/ymym3412/mlflow-docker-compose

FROM conda/miniconda3:latest

RUN mkdir -p /mlflow/mlruns

WORKDIR /mlflow

ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8

RUN echo "export LC_ALL=$LC_ALL" >> /etc/profile.d/locale.sh
RUN echo "export LANG=$LANG" >> /etc/profile.d/locale.sh

RUN apt-get update && apt-get install -y \
    build-essential \
    python3-dev \
    libpq-dev

RUN pip install -U pip && \
    pip install --ignore-installed google-cloud-storage && \
    pip install psycopg2 mlflow boto3

COPY ./mlflow_start.sh ./mlflow_start.sh
RUN chmod +x ./mlflow_start.sh

EXPOSE 80
EXPOSE 443

CMD ["./mlflow_start.sh"]

./mlflow_start.sh

#!/bin/bash

set -o errexit
set -o nounset
set -o pipefail

mlflow server \
    --backend-store-uri $DB_URI \
    --host 0.0.0.0 \
    --port 80 \
    --default-artifact-root ${S3_BUCKET}.env

docker-compose.yml

Where it is commented out, uncomment it when running in the gpu environment.

version: '3'
services:
  waitfordb:
    image: dadarek/wait-for-dependencies
    depends_on:
      - postgresql
    command: postgresql:5432

  postgresql:
    image: postgres:10.5
    container_name: postgresql
    ports:
      - 5432:5432
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: mlflow-db
      POSTGRES_INITDB_ARGS: "--encoding=UTF-8"
    hostname: postgresql
    restart: always

  mlflow:
    build:
      context: .
      dockerfile: Dockerfile_mlflow
    container_name: mlflow
    expose:
      - 80
      - 443
    ports:
      - "10006:80"
    depends_on:
      - postgresql
      - waitfordb
    environment:
      DB_URI: postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgresql:5432/mlflow-db
      VIRTUAL_PORT: 80
      S3_BUCKET: ${S3_BUCKET}
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}

  kedro:
    # runtime: nvidia
    build:
      context: .
      dockerfile: Dockerfile_kedro
    container_name: kedro
    environment:
      MLFLOW_TRACKING_URI: http://mlflow/
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      # NVIDIA_VISIBLE_DEVICES: all
    depends_on:
      - mlflow
    volumes:
      - ./:/app

build

Now that it's ready, let's build it!

$ docker-compose up

Workflow assembly

From here, we will work on the container (kedro).

After starting the container, execute the following command to enable working in the container.

$ docker-compose exec kedro /bin/bash
(base) root@hogehoge:/app#

Ok

What is Kedro

I will post only the information. It doesn't matter, but I thought it was "Kedoro", but it seems to have a completely different meaning.

--Repository: https://github.com/quantumblacklabs/kedro --Documentation: https://kedro.readthedocs.io/en/stable/ --pipeline comparison article: https://qiita.com/Minyus86/items/70622a1502b92ac6b29c

There is also a community, so if you have any problems, you should ask a question.

https://discourse.kedro.community/

kedro new

Prepare a project in which the workflow works.

(base) root@hogehoge:/app# kedro new
/opt/conda/lib/python3.7/site-packages/jinja2/utils.py:485: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import MutableMapping
/opt/conda/lib/python3.7/site-packages/jinja2/runtime.py:318: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Mapping
/opt/conda/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.2) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Project Name:
=============
Please enter a human readable name for your new project.
Spaces and punctuation are allowed.
 [New Kedro Project]: workflow

Repository Name:
================
Please enter a directory name for your new project repository.
Alphanumeric characters, hyphens and underscores are allowed.
Lowercase is recommended.
 [workflow]:

Python Package Name:
====================
Please enter a valid Python package name for your project package.
Alphanumeric characters and underscores are allowed.
Lowercase is recommended. Package name must start with a letter or underscore.
 [workflow]:

Change directory to the project generated in /app/workflow

A best-practice setup includes initialising git and creating a virtual environment before running `kedro install` to install project-specific dependencies. Refer to the Kedro documentation: https://kedro.readthedocs.io/

(base) root@hogehoge:/app# ls -l workflow/
total 12
-rw-r--r--  1 root root 4033 Dec 23 15:52 README.md
drwxr-xr-x  5 root root  160 Dec 23 15:52 conf
drwxr-xr-x 10 root root  320 Dec 23 15:52 data
drwxr-xr-x  3 root root   96 Dec 23 15:52 docs
drwxr-xr-x  4 root root  128 Dec 23 15:52 logs
drwxr-xr-x  3 root root   96 Dec 23 15:52 notebooks
-rw-r--r--  1 root root  341 Dec 23 15:52 pyproject.toml
-rw-r--r--  1 root root   47 Dec 23 15:52 setup.cfg
drwxr-xr-x  6 root root  192 Dec 23 15:52 src

Ok (Python version is 3.7, so it's a big deal)

kedro implementation overview

To give you a really rough idea, here are five files to play with.

--File to define - ~/workflow/conf/base/parameters.yml --File that defines the arguments of the process handled by node (described later) - ~/workflow/conf/base/catalog.yml --File that defines input data, generated intermediate data, evaluation data, etc. --Scripts that incorporate workflow - ~/workflow/src/workflow/nodes/ --Handle functions and classes that you want to process on the workflow as nodes --Processing can be implemented here, or module can be imported and logic can be implemented. ――This time, all the original processing is packed in the notebook, so salvage from the notebook is thrust into node - ~/workflow/src/workflow/pipelines/ ――It is a script that defines the assembly such as feeding the process you want to execute to node and feeding the result of nodeA to nodeB. - ~/workflow/src/workflow/hooks.py --Knead and assemble the prepared pipeline here ――By the way, the order of execution is that the input and output names are confirmed and the order is assembled when kedro is executed, so if you set different names, they may not be executed in order. Make sure that the output name of the previous pipeline and the input name of the next pipeline match.

parameters.yml

Defines the parameters to be passed to the function (node) to be executed on the workflow.

my_train_test_split:
  test_size: 0.2
  random_state: 71

catalog.yml

Define the data information to be handled in the workflow.

This time, I uploaded the image data zip on s3, so specify it.

You can also define data that is generated in the middle, such as intermediate data.

Normally, text.TextDataSet is supposed to be a text file etc., but since there is no type for the zip file (probably), I defined it so that binary data can be read and written using fs_args.

10_84_godzilla.zip:
  type: text.TextDataSet
  filepath: s3://godzilla-hogehoge-1234/workflow/data/01_raw/10_84.zip
  fs_args:
    open_args_load:
      mode: 'rb'
    open_args_save:
      mode: 'wb'

10_GMK_godizlla.zip:
  type: text.TextDataSet
  filepath: s3://godzilla-hogehoge-1234/workflow/data/01_raw/10_GMK.zip
  fs_args:
    open_args_load:
      mode: 'rb'
    open_args_save:
      mode: 'wb'

10_SOS_godzilla.zip:
  type: text.TextDataSet
  filepath: s3://godzilla-hogehoge-1234/workflow/data/01_raw/10_SOS.zip
  fs_args:
    open_args_load:
      mode: 'rb'
    open_args_save:
      mode: 'wb'

10_first_godzilla.zip:
  type: text.TextDataSet
  filepath: s3://godzilla-hogehoge-1234/workflow/data/01_raw/10_first.zip
  fs_args:
    open_args_load:
      mode: 'rb'
    open_args_save:
      mode: 'wb'

99_other.zip:
  type: text.TextDataSet
  filepath: s3://godzilla-hogehoge-1234/workflow/data/01_raw/99_other.zip
  fs_args:
    open_args_load:
      mode: 'rb'
    open_args_save:
      mode: 'wb'

classes_text:
  type: text.TextDataSet
  filepath: s3://godzilla-hogehoge-1234/workflow/data/01_raw/classes.txt

X_train:
  type: pickle.PickleDataSet
  filepath: s3://godzilla-hogehoge-1234/workflow/data/02_intermediate/X_train.pkl

X_test:
  type: pickle.PickleDataSet
  filepath: s3://godzilla-hogehoge-1234/workflow/data/02_intermediate/X_test.pkl

y_train:
  type: pickle.PickleDataSet
  filepath: s3://godzilla-hogehoge-1234/workflow/data/02_intermediate/y_train.pkl

y_test:
  type: pickle.PickleDataSet
  filepath: s3://godzilla-hogehoge-1234/workflow/data/02_intermediate/y_test.pkl

There are various types of datasets, so please check them.

https://kedro.readthedocs.io/en/stable/05_data/02_kedro_io.html

Furthermore, if you put the credentials in ~/workflow/conf/local/credentials.yml, you need to specify the key name in catalog.yml as well, but this time you have already specified it in .env. I don't need it because I have it.

node

Functions to be fed to node are prepared in ~/workflow/src/workflow/nodes /.

However, the place to write node is ~/workflow/src/workflow/pipelines /, so I will explain it in the next section.

Check github for the functions to be processed.

https://github.com/Aipakazuma/recognize-godzilla

pipeline

Yes. It is a pipeline. Let's actually assemble the pipeline when the function group that can be eaten by node is prepared.

Pre-processing assumes the order of (1) zip file expansion → (2) image reading → (3) training and test.

That's not the case

~/workflow/src/workflow/pipelines/preprocess_pipeline.py

from kedro.pipeline import node, Pipeline

from workflow.nodes.preprocess import unzip_image_data, load_image_data, my_train_test_split


def create_pipeline(**kwargs):
    return Pipeline(
        [
            node(
                func=unzip_image_data,
                inputs=[
                    '10_84_godzilla.zip',
                    '10_GMK_godizlla.zip',
                    '10_SOS_godzilla.zip',
                    '10_first_godzilla.zip',
                    '99_other.zip'
                ],
                outputs='unzip_path'
            ),
            node(
                func=load_image_data,
                inputs=[
                    'unzip_path',
                    'classes_text'
                ],
                outputs=[
                    'X',
                    'Y',
                    'classes'
                ]
            ),
            node(
                func=my_train_test_split,
                inputs=[
                    'X',
                    'Y',
                    'params:my_train_test_split'
                ],
                outputs=[
                    'X_train',
                    'X_test',
                    'y_train',
                    'y_test'
                ]
            ),
        ],
        tags=['preprocess'],
    )

I will assemble it like this.

Let's reduce the information a little and check it.

def create_pipeline(**kwargs):
    return Pipeline([
        node(),  # ①
        node(),  # ②
        node()   # ③
    ])

Something like this. I'm passing the node I want to execute to the Pipeline class as a list.

Now, let's check the node of the process of ①.

            node(
                func=unzip_image_data,
                inputs=[
                    '10_84_godzilla.zip',
                    '10_GMK_godizlla.zip',
                    '10_SOS_godzilla.zip',
                    '10_first_godzilla.zip',
                    '99_other.zip'
                ],
                outputs='unzip_path'
            ),

Specify the function you want to execute with func. inputs will be the argument of the specified function. In this case, you need to define five arguments for unzip_image_data. Also, the data corresponding to the name of inputs will be the data defined in catalog.yml. Kedro will automatically bind (?) While the character workflow is running.

outputs is the return value of the function. As with inputs, if there is data corresponding to catalog.yml, it will be saved as intermediate data. (If it is local, it is saved locally, if it is s3, it is saved in s3). If it does not support catalog.yml, it will be stored in memory by Kedro as kedro.io.MemoryDataSet during workflow execution.

However, I think there are some processes that have no return value, but honestly I'm still not sure what to do in that case (if anyone knows, please let me know).

Next, let's check the node of ②.

            node(
                func=load_image_data,
                inputs=[
                    'unzip_path',
                    'classes_text'
                ],
                outputs=[
                    'X',
                    'Y',
                    'classes'
                ]
            ),

You can see that the outputs of ① is included in the inputs of ②. It is not always necessary to set the output of the previous node to the next node, but Kedro assembles by associating the execution order of nodes with the names inputs and outputs, so if ① is executed before ② If you want to, implement it like ↑.

As a reminder, classes_text is defined in catalog.yml, so Kedro will load and prepare it for you during workflow execution.

The processing explained so far is a pipeline for pre-processing, but a separate file is prepared for model learning. If pipelines are prepared in multiple files, how to combine them is defined in ~/workflow/src/workflow/hooks.py.

hooks.py

Here is the definition that connects multiple pipelines.

Please note that, as explained above, the order in which nodes are executed is determined by the alignment of inputs and ouputs, so the order cannot be defined here.

~/workflow/src/workflow/hooks.py

class ProjectHooks:
    @hook_impl
    def register_pipelines(self) -> Dict[str, Pipeline]:
        """Register the project's pipeline.

        Returns:
            A mapping from a pipeline name to a ``Pipeline`` object.

        """
        pp = preprocess_pipeline.create_pipeline()  #For pretreatment
        tp = train_pipeline.create_pipeline()  #For training

        return {
            'preprocess': pp,
            'train': tp,
            '__default__': pp + tp
        }

The dictionary type is returned by return, but you can specify the pipeline you want to execute on the command line for key. It will be the key to use at that time.

As for __default__, as the name suggests, if you do not specify the pipeline name, the pipeline defined in __default__ will be executed.

Try to run

$ docker-compose exec godzilla_kedro /bin/bash -c 'cd /app/workflow; kedro run'

I was able to execute it. Let's check the product with s3.

TBD

Cooperation with Mlflow

TBD

What is hook

Try to work with mlflow using hook

Check the result

Try using Kedro

Points that look good

--Data linkage is easy --If you get used to Python module development, you can easily create a workflow. ――It's okay, you can hit the notebook! --Easy to introduce data catalog version control (?)

I don't know the correct answer, but I was able to implement it for the time being

--When you want to branch node by parameter TBD --Read catalog.yml and want to retrieve the key TBD

I want to change here as well, but it seems difficult unless I mess with the inside

--I want to change the versioning directory structure --I want to simplify the data specification of --load-version ――If you have more than one, you have to do more than one. ――I don't know if the version is loaded properly in the log. --Is there a binary data catalog type? ?? ?? --No catalog type for pytorch (maybe)

Miscellaneous feelings

MLOps is fun! !! From now on, cancer b ... tejo ... uho ... u ... (˘ω˘)

Recommended Posts

Let's make Godzilla's image recognition model preprocessing, learning and deployment feel good
Let's make an image recognition model with your own data and play!
Deep learning image recognition 2 model implementation
Image recognition model using deep learning in 2016
Deep learning image recognition 3 after model creation
Implementation of Deep Learning model for image recognition
Real-time image recognition on mobile devices with TensorFlow learning model
Deep learning image recognition 1 theory
Learning model creation, learning and reasoning
About the shortest path to create an image recognition model by machine learning and implement an Android application