Environment construction of keras and tensorflow, jupyter lab by docker and connection to jupyter by port forwarding

Overview

Build keras gpu and jupyterlab environment with docker. I will omit the installation of docker to see what docker is. In addition, I will briefly explain port forwarding.

Image pull

Pull the image with the latest gpu tag in dockerhub of tensorflow. If the tag is only latest, cuda cannot be used. Here's what to do

$ sudo docker pull tensorflow/tensorflow:latest-gpu

Check the image

$ sudo docker images
REPOSITORY                        TAG                 IMAGE ID            CREATED             SIZE
tensorflow/tensorflow             latest-gpu          20fd065e3887        7 weeks ago         3.15GB

Install modules and jupyter that you may need

Module to install

This time I will import

• matplotlib
• cv2 (opencv-python)
• keras
• jupyter lab

Create a docker container from the image before importing. Enter as root when building the environment.

$ sudo docker run -it tensorflow/tensorflow:latest-gpu

pip and apt updates

# pip install --upgrade pip
# apt-get update -y

Preparing to install opencv

See below Building python TensorFlow and OpenCV execution environment with Docker

# apt-get install -y libopencv-dev

It takes a lot of time.

Installation of required modules

# pip install matplotlib opencv-python keras
# pip install jupyterlab

Test if the module can be imported

# python
Python 3.6.9 (default, Jul 17 2020, 12:50:27)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib
>>> import numpy
>>> import cv2
>>> import keras
2020-09-20 01:02:34.313977: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
>>> import tensorflow

It looks okay. You can exit interactive mode with Ctrl + d. I have jupyterlab installed with this, but I have to do port forwarding to run it. If you want to see the detailed execution procedure, please read this article until the end.

Image commit

Commit the container to the image before testing the jupyter lab. In other words, it saves the current status of various installations. Exit the container with Ctrl + p, Ctrl + q running.

Checking currently running containers

$ sudo docker ps -n=-1
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                  NAMES
a24503258ede        tensorflow/tensorflow:latest-gpu   "/bin/bash"              24 minutes ago      Up 24 minutes                              romantic_gauss

Container commit

$ sudo docker commit a24503258ede keras-jupyter

sudo docker commit (container ID) (name of the image you want to give). After that, if you build the image named keras-jupyter, the container you made so far will be completed.

Stop container

Stop the container once.

$ sudo docker stop a245
$ sudo docker ps -n=-1
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES

The container ID responds with about 4 characters at first.

About port forwarding

I will check the operation of jupyterlab from now on, but I will briefly explain port forwarding.

What is port forwarding?

Currently, I'm trying to ssh connect to the server from my personal computer as shown in the figure below, run docker on it, and start jupyterlab on it. However, since jupyterlab is handled on the browser, you have to create another connection route to jupyterlab in addition to the ssh connection.

Therefore, creating a connection route (tunnel) other than ssh so that you can connect to jupyter even on the browser is called ** port forwarding **. You can open a tunnel in addition to ssh port 22 by adding options to each command as shown in the figure below.

However, this time I changed all the port numbers for explanation, but it would be better to unify all the port forwarding ports to 8888 etc. so as not to make a mistake. Now, let's check if port forwarding is possible through the jupyterlab test.

Port forwarding to jupyterlab

port forward with ssh

First, connect to the server locally with ssh.

$ ssh [email protected] -L 8800:x.x.x.x:8880

Add options such as -L {local port number}: {server ip}: {port number you want to connect with the server}.

port forward with docker

Add -p 8880: 8888 to the options. This will connect server port 8880 to docker 8888 when you access docker from the server.

$ sudo docker run -it -p 8880:8888 keras-jupyter

This led the local number 8800 to docker number 8888.

Start jupyterlab

Next, start jupyterlab on the container with the following command.

# jupyter lab --port 8888 --ip=* --allow-root

Explanation of the following commands

• First, docker port8888 is connected to the local, so jupyter port is set to 8888. ( --port 8888)
• If you start jupyter as it is, you cannot connect because you do not know the ip, so connect to the docker environment and ad hoc. (--ip = *)
• Finally, since docker is currently started as root, it allows connection by root. ( --allow-root)

The output is as follows``http://127.0.0.1:8888/?token=614f6e5d410a1c459baa691ee28fdd0e2593c4644189a09e```Copy.

[W 02:52:43.023 LabApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
[I 02:52:43.028 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.6/dist-packages/jupyterlab
[I 02:52:43.028 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 02:52:43.030 LabApp] Serving notebooks from local directory: /
[I 02:52:43.030 LabApp] Jupyter Notebook 6.1.4 is running at:
[I 02:52:43.030 LabApp] http://0829a84d25eb:8888/?token=614f6e5d410a1c459baa691ee28fdd0e2593c4644189a09e
[I 02:52:43.030 LabApp]  or http://127.0.0.1:8888/?token=614f6e5d410a1c459baa691ee28fdd0e2593c4644189a09e
[I 02:52:43.030 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

###Connection to jupyter Change a part of the url as below and paste it into your local browser.

http://localhost:8800/?token=614f6e5d410a1c459baa691ee28fdd0e2593c4644189a09e

You can now connect to the jupyter running on the server by port forwarding.

##Digression When actually executing docker, binding the user with docker and server is not explained in detail here, but it is better to register the docker run command in alias and use it as follows.

alias docker-run-keras = 'sudo docker run -v /etc/group:/etc/group:ro -v /etc/passwd:/etc/passwd:ro -v $HOME:$HOME -u $(id -u $USER):$(id -g $USER) -it --gpus 2 -p 8888:8888 --shm-size 30g keras-jupyter