In this article, we will use Docker to build an environment for tensorflow2.3 + Python3.8. You can easily build an environment by rewriting the contents of this article to the corresponding CUDA and Python versions of any TensorFlow or PyTorch version. Since there are a mixture of articles related to building a GPU environment with Docker, I will organize a memorandum of what I did as of October 2020.
In this article, I installed Python3.8 and Poetry based on the nvidia / cuda: 10.1-cudnn7-devel-ubuntu18.04 image and used Poetry to run tensorflow2.3 in a virtual environment. Install it.
--Docker beginners --People who want to build an environment for multiple framework versions such as TensorFlow on one machine
The following 1 ~ 3 seems to be easier to do in this article (I have not tried it myself) [What's going on with NVIDIA Docker now? (20.09 version)](https://medium.com/nvidiajapan/nvidia-docker-%E3%81%A3%E3%81%A6%E4%BB%8A%E3%81%A9%E3%81%86 % E3% 81% AA% E3% 81% A3% E3% 81% A6% E3% 82% 8B% E3% 81% AE-20-09-% E7% 89% 88-558fae883f44)
Check the GPU type (check that there is one made by NVIDIA).
$ lspci | grep VGA
03:00.0 VGA compatible controller: NVIDIA Corporation Device 1b80 (rev a1)
04:00.0 VGA compatible controller: NVIDIA Corporation Device 1b80 (rev a1)
If you already have the NVIDIA driver installed, remove it.
$ dpkg -l | grep nvidia-
$ sudo apt-get purge nvidia-*
Add the driver repository.
$ sudo add-apt-repository ppa:graphics-drivers/ppa
$ sudo apt update
Check the recommended driver.
$ sudo ubuntu-drivers devices
== /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.0 ==
modalias : pci:v000010DEd00001B80sv00001462sd0000336Bbc03sc00i00
vendor : NVIDIA Corporation
driver : nvidia-430 - third-party free recommended
driver : nvidia-415 - third-party free
driver : xserver-xorg-video-nouveau - distro free builtin
Install the recommended nvidia-430.
$ sudo apt-get install nvidia-driver-430
$ sudo ubuntu-drivers autoinstall
Reboot the OS.
$ sudo reboot
Check if the NVIDIA driver is installed successfully.
$ nvidia-smi
Thu Oct 15 16:49:51 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.64 Driver Version: 430.64 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1080 Off | 00000000:03:00.0 Off | N/A |
| 28% 40C P8 10W / 180W | 2MiB / 8119MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 1080 Off | 00000000:04:00.0 On | N/A |
| 28% 41C P8 13W / 180W | 533MiB / 8085MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 1 1247 G /usr/lib/xorg/Xorg 422MiB |
| 1 1583 G compiz 108MiB |
+-----------------------------------------------------------------------------+
You can install Docker according to Official Document. The following is a copy and paste of a part of the official Document, so please read the Document for details.
SET UP THE REPOSITORY 1. Update the apt package index and install packages to allow apt to use a repository over HTTPS:
$ sudo apt-get update
$ sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common
2. Add Docker’s official GPG key:
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo apt-key fingerprint 0EBFCD88
pub rsa4096 2017-02-22 [SCEA]
9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88
uid [ unknown] Docker Release (CE deb) <[email protected]>
sub rsa4096 2017-02-22 [S]
3. Use the following command to set up the stable repository.
$ sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
INSTALL DOCKER ENGINE 1. Update the apt package index, and install the latest version of Docker Engine and containerd, or go to the next step to install a specific version:
$ sudo apt-get update
$ sudo apt-get install docker-ce docker-ce-cli containerd.io
3. Verify that Docker Engine is installed correctly by running the hello-world image.
$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete
Digest: sha256:8c5aeeb6a5f3ba4883347d3747a7249f491766ca1caa47e5da5dfcf6b9b717c0
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
...
Check if Docker is installed correctly.
$ dpkg -l | grep -e docker -e containerd.io
ii containerd.io 1.3.7-1 amd64 An open and reliable container runtime
ii docker-ce 5:19.03.13~3-0~ubuntu-xenial amd64 Docker: the open-source application container engine
ii docker-ce-cli 5:19.03.13~3-0~ubuntu-xenial amd64 Docker CLI: the open-source application container engine
$ sudo docker version
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:59 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.13
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:01:30 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.7
GitCommit: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest bf756fb1ae65 9 months ago 13.3kB
[Optional] Allows docker commands to be executed without sudo.
$ sudo usermod -aG docker $USER
With the September 2020 update, it seems that all you have to do is install nvidia-docker2. ([What's happening with NVIDIA Docker now? (20.09 version)](https://medium.com/nvidiajapan/nvidia-docker-%E3%81%A3%E3%81%A6%E4%BB%8A % E3% 81% A9% E3% 81% 86% E3% 81% AA% E3% 81% A3% E3% 81% A6% E3% 82% 8B% E3% 81% AE-20-09-% E7% 89% 88-558fae883f44)))
(If you install nvidia-docker2, nvidia-container-toolkit will be installed as well)
$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
$ sudo apt-get update
$ sudo apt-get install -y nvidia-docker2
$ sudo systemctl restart docker
Write the following contents in Dockerfile_gpu.
Select the version of CUDA and cuDNN according to the version of the framework you want to install.
Here, according to tensorflow2.3, [Image of CUDA10.1 + cuDNN7](https://hub.docker.com/r/nvidia/cuda/tags?page=1&name=10.1-cudnn7-devel-ubuntu18. Based on 04).
Dockerfile_gpu
FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends tzdata
RUN apt-get update && apt-get install -y --no-install-recommends \
wget \
curl \
make \
build-essential \
libssl-dev \
zlib1g-dev \
libbz2-dev \
libreadline-dev \
libsqlite3-dev \
llvm \
libncurses5-dev \
libncursesw5-dev \
xz-utils \
tk-dev \
libffi-dev \
liblzma-dev \
vim \
graphviz
ENV TZ Asia/Tokyo
WORKDIR /root/
RUN wget https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tar.xz \
&& tar xvf Python-3.8.5.tar.xz \
&& cd Python-3.8.5 \
&& ./configure --enable-optimizations \
&& make install
RUN rm Python-3.8.5.tar.xz
WORKDIR /root/Python-3.8.5
RUN ln -fs /root/Python-3.8.5/python /usr/bin/python
RUN curl -kL https://bootstrap.pypa.io/get-pip.py | python
RUN rm -rf /var/lib/apt/lists/*
RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
ENV SHELL /bin/bash -l
ENV POETRY_CACHE /work/.cache/poetry
ENV PIP_CACHE_DIR /work/.cache/pip
RUN $HOME/.poetry/bin/poetry config virtualenvs.path $POETRY_CACHE
ENV PATH ${PATH}:/root/.poetry/bin:/bin:/usr/local/bin:/usr/bin
CMD ["bash", "-l"]
Write the following contents in Makefile (rewrite ʻIMAGE_NAME` as appropriate).
Makefile
FOLDER=$$(pwd)
IMAGE_NAME=test:latest
.PHONY: build-gpu
build-gpu: # Build docker image
echo "Building Dockerfile"
docker build -t ${IMAGE_NAME} . -f Dockerfile_gpu
.PHONY: start-gpu
start-gpu: build-gpu # Start docker container
echo "Starting container ${IMAGE_NAME}"
docker run --gpus all --rm -it -v ${FOLDER}:/work -w /work ${IMAGE_NAME}
Build a Docke image to create a Docker container.
$ make start-gpu
Install tensorflow-gpu 2.3 with Poetry.
# poetry init
# poetry add tensorflow-gpu==2.3
Execute any file.
# poetry run python file_name.py
In this article, I have summarized the procedure for building an environment for tensorflow2.3 + Python3.8 using Docker in a memorandum. It's convenient to have only the NVIDIA driver, Docker, and NVIDIA Container Toolkit installed on the host OS. In this article, I installed TensorFlow with Poetry, but of course you can pip install with Dockerfile.
Troubleshooting apt update Google Chrome related errors https://sicklylife.hatenablog.com/entry/2017/08/08/193118
"could not open" list file due to "permission denied" error https://askubuntu.com/a/1146593
nouveau Disable nouveau https://qiita.com/kawazu191128/items/8a46308be6949f5bda57#nouveau%E7%84%A1%E5%8A%B9%E5%8C%96
apt-get install nvidia-driver-300 If UEFI Secure Boot is enabled, you need to disable Secure Boot or set a key.
Temporary failure resolving '....com' https://github.com/moby/moby/issues/5779#issuecomment-323433618
https://mzryuka.hatenablog.jp/entry/2019/08/07/235613
Recommended Posts