[DOCKER] Build a Python machine learning environment with a container

Build a Python machine learning environment with a container

Build a Python machine learning environment in a container on your laptop.

This time, we will build a container that can use fastText and Python3 mainly as an environment for text-based machine learning.

image

--For execution (learning and verification), log in to the machine learning container with a shell and execute fastText and Python3. --Mount the source code and learning files on the folder on your PC. --The editor uses Visual Studio Code.

Edit

work

https://qiita.com/penpenta/items/3b7a0f1e27bbab56a95f

environment

Launch the base container image

First, start the base container image. It is better to keep the execution command here the same as the command to execute the built container image. This time we will build fastText inside the container, so we will use the CentOS image.

  1. Execute the base command with the following command.
docker run -it -v /c/temp/data:/data --rm centos:centos8 /bin/bash

Install and verify what you need little by little

You can build with Dockerfile from the beginning, but you may get an installation error. This work is done to gradually verify the environment.

Python installation-fastText build

  1. Execute the following command. * Wait for a while with "Running setup.py install for fasttext ..." after executing pip3 install.
dnf -y install python36
yum install -y git make gcc gcc-c++
yum install -y python36-devel
cd /usr/local/src
git clone https://github.com/facebookresearch/fastText.git
cd fastText
pip3 install .
  1. Check if it can be imported as a python package. Execute the following command and if there are no errors, it is OK.
python3
import fasttext
exit()

  1. Save the container you have worked on with docker export. From the next time onwards, you can continue working from here with docker import.
docker export {Image ID} > {file name}.tar

Install machine learning package

Install the standard package.

  1. Execute the following command.
pip3 install numpy pandas matplotlib scikit-learn

Install Japanese processing package

Install the packages required for Japanese processing. The dictionary is not required, so you can skip it.

  1. Execute the following command.
rpm -ivh http://packages.groonga.org/centos/groonga-release-1.1.0-1.noarch.rpm
yum -y makecache
yum -y install mecab mecab-ipadic
yum -y install --nogpgcheck mecab-devel
pip3 install mecab-python3 neologdn emoji
  1. Install the dictionary (install diffutils patch which file openssl as it is required by the installation script)
yum install -y diffutils patch which file openssl

cd /usr/local/src
git clone --depth 1 https://github.com/neologd/mecab-ipadic-neologd.git
cd mecab-ipadic-neologd
./bin/install-mecab-ipadic-neologd -n -y

Export environment

This time I want to use it for personal use and always use the new package version, so I will export the environment to the work Dockerfile so far.

--Points when creating a Dockerfile --Use WORKDIR instead of cd (cd cannot be used in Dockerfile) --The command should be executed by bash. --The dictionary is commented out. (Because it takes time to build)

Dockerfile

FROM centos:centos8

SHELL ["/bin/bash", "-c"]
#Python Install

RUN dnf -y install python36

#fastText Build

RUN yum install -y git make gcc gcc-c++
RUN yum install -y python36-devel
WORKDIR /usr/local/src
RUN git clone https://github.com/facebookresearch/fastText.git
WORKDIR /usr/local/src/fastText
RUN pip3 install .

#Install Python Package
RUN pip3 install numpy pandas matplotlib scikit-learn
RUN rpm -ivh http://packages.groonga.org/centos/groonga-release-1.1.0-1.noarch.rpm
RUN yum -y makecache
RUN yum -y install --nogpgcheck mecab mecab-ipadic mecab-devel
RUN pip3 install mecab-python3 neologdn emoji

#Install Mecab Dictionary

# RUN yum install -y diffutils patch which file openssl
# 
# WORKDIR /usr/local/src
# RUN git clone --depth 1 https://github.com/neologd/mecab-ipadic-neologd.git
# WORKDIR /usr/local/src/mecab-ipadic-neologd
# RUN ./bin/install-mecab-ipadic-neologd -n -y

Build and boot confirmation

  1. Build using Dockerfile.
docker build -t fasttext/centos8:v1.0 .
  1. Confirm that it can be started with the following command.
docker run -it -v /c/temp/data:/data --rm fasttext/centos8:v1.0 /bin/bash

Summary

By following the steps below, you have built a Python machine learning environment in a container on your laptop.

  1. Launch the base container image
  2. Install and verify what you need little by little
  3. Export the container image

With remote work, it was not possible to always connect to the in-house VPN, and development work on the development server was difficult, but this environment solved it.

Since it takes time to learn on a notebook PC, there are problems such as "resources are occupied for learning" and "cannot shut down", but if the same container is loaded in advance on the development server, "less on a notebook PC" You can solve this problem by following the procedure "Check operation with epoch → Increase epoch on development server and learn". It is also possible to hook the commit of the source code and automatically run the learning on the development server. I will summarize the article separately about that.

Recommended Posts

Build a Python machine learning environment with a container
Build a machine learning application development environment with Python
Build a machine learning environment
Build a machine learning Python environment on Mac OS
Build a python machine learning study environment on macOS sierra
Build a machine learning scikit-learn environment with VirtualBox and Ubuntu
Build a python virtual environment with pyenv
Build a modern Python environment with Neovim
Building a Windows 7 environment for getting started with machine learning with Python
Build a C language development environment with a container
Build a python environment with ansible on centos6
[Python] Build a Django development environment with Docker
Build a Python environment with OSX El capitan
Build a python execution environment with VS Code
Build python3 environment with ubuntu 16.04
Build python environment with direnv
Build a Python environment offline
Beginning with Python machine learning
Build an interactive environment for machine learning in Python
Run a machine learning pipeline with Cloud Dataflow (Python)
Build a python virtual environment with virtualenv and virtualenvwrapper
Memo for building a machine learning environment using Python
Create a machine learning environment from scratch with Winsows 10
Build a machine learning environment natively on Windows 10 (x64)
Build python virtual environment with virtualenv
Machine learning with python (1) Overall classification
Build Mysql + Python environment with docker
Create a virtual environment with Python!
Building a virtual environment with Python 3
Build a python3 environment on CentOS7
"Scraping & machine learning with Python" Learning memo
Build a machine learning environment on mac (pyenv, deeplearning, opencv)
Build an environment for machine learning using Python on MacOSX
(Now) Build a GPU Deep Learning environment with GeForce GTX 960
[Django] Build a Django container (Docker) development environment quickly with PyCharm
How to build a python2.7 series development environment with Vagrant
Create a python machine learning model relearning mechanism with mlflow
Machine learning environment settings based on Python 3 on Mac (coexistence with Python 2)
Build a PyData environment for a machine learning study session (January 2017)
Build a Python environment with WSL + Pyenv + Jupyter + VS Code
Until you create a machine learning environment with Python on Windows 7 and run it
I tried to build an environment for machine learning with Python (Mac OS X)
[Pyenv] Building a python environment with ubuntu 16.04
Python & Machine Learning Study Memo: Environment Preparation
Building a Python3 environment with Amazon Linux2
Easily build a development environment with Laragon
Build Jupyter Lab (Python) environment with Docker
Amplify images for machine learning with python
Build a python environment on MacOS (Catallina)
Build a Tensorflow environment with Raspberry Pi [2020]
Machine learning with python (2) Simple regression analysis
A story about machine learning with Kyasuket
Building a Python 3.6 environment with Windows + PowerShell
I want to build a Python environment
Build a Fast API environment with docker-compose
[Shakyo] Encounter with Python for machine learning
Build Python environment with Anaconda on Mac
Python and machine learning environment construction (macOS)
Build a Python + OpenCV environment on Cloud9
Creating a development environment for machine learning
[Linux] Build a Docker environment with Amazon Linux 2