Preparing the execution environment of PyTorch with Docker November 2019

Overview

--Prepare the execution environment of Deep Learning by PyTorch with Docker container --Docker version 19.03 or later, if you have the GPU driver of the host OS and nvidia-container-runtime, the others will be confined in the container ――In most cases, you can reproduce the environment with a single command.

Preparation

Roughly the following preparations are required.

--docker version 19.03 or later -See Official Documentation (this link is for Ubuntu)

Create a Dockerfile

CUDA and CuDNN are stored in the container, and the GPU Driver on the host side is used. In this Dockerfile, Python 3.7.1 and the Python package are installed based on the image containing CUDA: 10.1 and CuDNN: 7 in ʻUbuntu: 18.04`.

FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
 
WORKDIR /code
ENV PYTHON_VERSION 3.7.1
ENV HOME /root
ENV PYTHON_ROOT $HOME/local/python-$PYTHON_VERSION
ENV PATH $PYTHON_ROOT/bin:$PATH
ENV PYENV_ROOT $HOME/.pyenv
ENV TZ=Asia/Tokyo
ADD requirements.txt /code
RUN export DEBIAN_FRONTEND=noninteractive && \
    apt-get -y update && \
    apt-get -y upgrade && \
    apt-get -y install tzdata
RUN apt-get -y install git make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev && \
     apt-get -y install wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev && \
    git clone https://github.com/pyenv/pyenv.git $PYENV_ROOT && \
    $PYENV_ROOT/plugins/python-build/install.sh && \
    /usr/local/bin/python-build -v $PYTHON_VERSION $PYTHON_ROOT && \
    rm -rf $PYENV_ROOT && \
    pip install -r requirements.txt

An example of the contents of requirements.txt

requirements.txt


torch
torchvision

Use container environment

First, build the container. Execute the following command in the directory where the Dockerfile is created.

docker build . -t torch3.7

Enter the container and check if the GPU can be recognized. If the following display appears with the nvidia-smi command, it is ok.

$ docker run -it --gpus all torch3.7 bash
$ nvidia-smi
 Tue Nov 19 15:01:12 2019       
 +-----------------------------------------------------------------------------+
 | NVIDIA-SMI 430.50       Driver Version: 430.50       CUDA Version: 10.1     |
 |-------------------------------+----------------------+----------------------+
 | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
 | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
 |===============================+======================+======================|
 |   0  GeForce GTX 166...  Off  | 00000000:1F:00.0 Off |                  N/A |
 | 45%   36C    P0     1W / 120W |      0MiB /  5944MiB |      0%      Default |
 +-------------------------------+----------------------+----------------------+
                                                                                
 +-----------------------------------------------------------------------------+
 | Processes:                                                       GPU Memory |
 |  GPU       PID   Type   Process name                             Usage      |
 |=============================================================================|
 |  No running processes found                                                 |
 +-----------------------------------------------------------------------------+

Finally, confirm that PyTorch's torch.Tensor can be loaded in the memory on the GPU. I was able to use torch.Tensor on the GPU.

$ python
>>> import torch
>>> torch.cuda.is_available()
True
>>> x = torch.Tensor([0, 1])
>>> x
tensor([0., 1.])
>>> x.cuda()
tensor([0., 1.], device='cuda:0')

By the way ...

It seems that volume is often used synchronously between the host and the container. In that case, change the following command as appropriate and use it.

docker run -it -v /path/to/your_project_directory:/code --gpus all torch3.7 bash

Recommended Posts

Preparing the execution environment of PyTorch with Docker November 2019
Prepare the execution environment of Python3 with Docker
Note: Prepare the environment of CmdStanPy with docker
Build PyPy execution environment with Docker
Build the execution environment of Jupyter Lab
Build PyPy and Python execution environment with Docker
The story of sharing the pyenv environment with multiple users
Example of pytest environment to fix database with Docker
Implement PyTorch + GPU with Docker
Prepare python3 environment with Docker
Prediction of Nikkei 225 with Pytorch 2
Prediction of Nikkei 225 with Pytorch
Specify the project name of docker-compose with Docker integration of Pycharm
Save the output of GAN one by one ~ With the implementation of GAN by PyTorch ~
Get the host name of the host PC with Docker on Linux
The day of docker run (note)
Build Mysql + Python environment with docker
Take the execution log of Celery
Prediction of Nikkei 225 with Pytorch ~ Intermission ~
How to monitor the execution status of sqlldr with the pv command
pytorch @ python3.8 environment construction with pipenv
Create a C ++ and Python execution environment with WSL2 + Docker + VSCode
Rebuild Django's development environment with Docker! !! !! !!
Data science environment construction with Docker
Periodically log the value of Omron environment sensor with Raspberry Pi
Prepare the development environment with anyenv
Environment construction of Flask / MySql / Apache / mod_wsgi / virtualenv with Redhat7 (Python2.7) November 2020
I set the environment variable with Docker and displayed it in Python
Prepare the environment of Chainer on EC2 spot instance with AWS Lambda
Logging the value of Omron environment sensor with Raspberry Pi (USB type)
Access the Docker Remote API with Requests
I checked the contents of docker volume
Run the IDCF cloud CLI with Docker
Build Jupyter Lab (Python) environment with Docker
Align the size of the colorbar with matplotlib
Behind the flyer: Using Docker with Python
Check the existence of the file with python
About the virtual environment of python version 3.7
Use of virtualenv, Python's independent execution environment
Get a local DynamoDB environment with Docker
The third night of the loop with for
[Linux] Build a jenkins environment with Docker
Launch environment with LineBot + Heroku + Docker + Python
The second night of the loop with for
Count the number of characters with echo
Build NGINX + NGINX Unit + MySQL environment with Docker
[Linux] Build a Docker environment with Amazon Linux 2
I installed Pygame with Python 3.5.1 in the environment of pyenv on OS X
[Verification] Try to align the point cloud with the optimization function of pytorch Part 1
The story of doing deep learning with TPU
Hello World with gRPC / go in Docker environment
Build Django + NGINX + PostgreSQL development environment with Docker
Analytical environment construction with Docker (jupyter notebook + PostgreSQL)
Build the fastest Django development environment with docker-compose
Execution environment on the Web by "Project Jupyter"
Create execution environment for each language with boot2docker
Setting to output the log of cron execution
[Note] Export the html of the site with python.
See the behavior of drunkenness with reinforcement learning
Go (Echo) Go Modules × Build development environment with Docker
Increase the font size of the graph with matplotlib