Put NVIDIA + Ubuntu + Docker + Tensorflow + ROS2 etc. in the first PC and make it as Mac-like as possible [Updating]

I. What I want to do

――Get a desktop PC with NVIDIA GPU for about 10 Sakichi --Install Ubuntu20.04 and adjust the usability to Mac --Install Docker on Ubuntu 20.04 and build the following system on Docker - TensorFlow+Keras --ROS2 (Not listed. Additional updates are ongoing) --PyTorch (Not listed. Additional updates are ongoing)

Ⅰ.Ⅰ. Match the usability to Mac? </ summary>

  1. Change "Left Neighbor Key (Super / alt) Key" of "Space Key" to "Left Neighbor Ctrl Key" and press "Left Neighbor Ctrl Key" + "X / C / V" to "Cut / Copy / Paste" (Same usage as "Left ⌘ key" on Mac): OK
  2. Enable "English / Japanese" input source switching with "Space key" and "Ctrl key on the left" (same usage as input source switching with US key on Mac): OK
  3. Enable conversion to "half-width katakana / half-width English / hiragana / full-width katakana / full-width English" with "Option key" + "a / s / z / x / c" (Kotoeri basic settings): OK
  4. Change "Capslock key" to "Ctrl key" (personal hobby): OK
  5. I want to use "TrackPad" for input (because I have only used Trackpad for the last 20 years): NG

The result was 4 wins and 1 loss

Ⅱ. Background </ summary>

I installed "Ubuntu 18.04" on a MacBookPro + virtual environment and ran "DeepLeaning / ROS1" on it to play, but ** [Seriously start "DeepLeaning" & "Make from scratch DeepLeaning②" Seems to need a GPU] **, so I was looking for ** "eGPU" ** that connects to the MacBook Pro.

"NVIDIA GPU" is required for "Deep Leaning"

Information with! ** Apple and NVIDIA have been in a fight all the time. ** However, I was disappointed to buy a PC (I had a small amount of money), so I tried various things, but

--Microsoft Azure: Linux beginners are heavily charged while going back and forth --Google Colab: 90 minutes timeout --The reality that GPUs don't work well in virtual environments (CDLE (Community of Deep Learning Evangelist) members who taught me, thank you)

From the result of, I was angry that I would use it in ROS, and decided to buy a PC with a GPU dedicated to Ubuntu (at that time, for the first time in my life, I realized the reality of buying a PC other than Mac, and my experience value was low. I'm a little scared)

Ⅲ. Flow until the end of the mission

  1. Secure hardware
  2. Ubuntu installation and NVIDIA driver operation check
  3. Mount the internal HDD (I want to use SSD for OS and HDD for Data separately)
  4. Mac-like basic settings and keyboard settings
  5. Installation of basic software
  6. Installation of basic development software
  7. Install Docker (for NVIDIA)
  8. Build an environment of [TensorFlow + Keras] on Docker
  9. Build [ROS2] environment on Docker (additional updates are ongoing)
  10. Build an environment for [PyTorch] on Docker (additional updates are ongoing)

Ⅳ. Mission start

1. Secure hardware </ summary>

1.1. PC body (for beginner level of Deep Leaning)

Since I have never purchased any hardware other than Mac, I consulted with "CDLE" Slack and decided to purchase a BTO gaming PC with specifications close to the recommended specifications at the Net shop (the gaming PC always has a GPU). From the advice that total cost performance is good)

item Consultation result reality
CPU Core i7 or i5 Gen10 or above i5 Gen9
memory As much as possible 16GB
GPU About GeForce GTX1600 SUPER GeForce GTX1600 SUPER
SSD (For OS) 250GB x 2(Ubuntu and Win Dual Boot) 480GB x 1(Ubuntu only)
HDD (For Data) - 2TB

The difference between the recommendation and the reality is the result of a compromise between "differences in the basic model of each shop" and "budget". In the end, it was about 13 Sakichi.

** [Teaching from CDLE members] ** ** ・ The PC for Deep Leaning says "Emphasis is placed on the GPU and memory size on the GPU", but in actual operation, the preprocessing CPU and memory often become bottlenecks, so a well-balanced configuration is possible. desirable**

** ・ BTOPC can change CPU / memory / GPU to some extent later, so it is better to purchase according to your budget first and try and make an error to see if it suits your needs and change the configuration. ** **

** [Trouble] ** One month after the purchase, the phenomenon that "If you do not remove the memory from the motherboard once, it will not start with a probability of 80%" occurs. When I consulted with the Net shop I purchased,

--Send the PC back at your own expense --Bring it to a physical store

, 2 choices. Fortunately, there was a real store in the neighborhood, so I brought it in and was hospitalized for 4 days, then discharged by memory exchange. In addition, I will also teach you how to isolate the startup status by the LED color of "ASUS Mother Board" I was quite impatient, so it was really helpful to have a real store nearby. I will buy it at that store next time

【Lesson】 ** ・ Cheapness and specifications are important for BTO PC, but beginners should choose a purchaser in consideration of troubleshooting **

1.2. Input system

――Since the work space is small, we built everything wirelessly.

1.2.1. PC side: Bluetooth USB dongle

--Newiy Start Ver4.0 Adapter: 1 → 2

--I bought it because Amazon's product description said that it could be used on Linux. It did work, but the mouse and keyboard opponents seemed to be awkward with one, and sometimes the mouse cursor flew, so the operation became strange, so I purchased an additional one and changed it to two (but sometimes the mouse becomes heavy. The cause is the PC side ?)

1.2.2. Mouse (?)

--Apple: Magic TrackPad (first generation) → LogiCool: M337

--Get the Magic TrackPad (first generation) with Yahoo Auction because you want to match the input system to the MacBook.

-** [Trouble] ** ――The multi-touch feels good with a very good software called "fusuma", but the movement of the cursor itself is unpleasant, and I repeated various "google + setting + restart" for a full two days, but it does not fit well On the contrary, the stress is just getting higher. I gave up and bought it for Logitech's cheap "Blurtooth Mouse". --Unexpectedly, the mouse buttons (left and right scroll / navigation buttons) are also key-assigned on Ubuntu, and I regret that I should have made it a "mouse with a few more buttons".

1.2.3. Keyboard

--Apple wireless keyboard (US) → To be determined

--Apple wireless keyboard (US) looks great alongside Apple: Magic TrackPad (first generation), so get it at Yahoo! Auctions. The key is the familiar US key (Old Macs didn't have the JIS key itself, so you can still only use the US key)

-** [Trouble] ** ――It's working relatively well, but sometimes (sleep / when you change the OS settings and restart) the connection is lost. If it runs out, you need to connect a wired keyboard, set up the Bluetooth connection, and restart it before you can use it. I'm in a lot of trouble, so [Would you like to buy a new one? ] Is being considered (although the key touch is good) ――As an aside, I changed my company notebook PC to a US key accordingly.

【Lesson】 **-Apple products cannot easily obtain the same Touch & Feel as Apple products just by connecting to Linux ** **-Emergency wired keyboard and mouse are required. Prepared within reach **

2. Ubuntu installation and NVIDIA driver operation check </ summary>

2.1. Preparing the installation image of Ubuntu 20.04 (Japanese version)

[What to do after installing Ubuntu 20.04](https://ohmyenter.com/things-to-do-after-ubuntu-20-04-installation/)

--Ubuntu downloads the 20.04 image file of Ubuntu Japan Official Website ――There are many detailed pages around here, so I will hand them over to them. Please try to find it according to your installation environment. ――I had a "DVD drive" on my PC, so I burned the image file to a live DVD. --Burn ubuntu image file to DVD as it is and you're done

2.2. Installing Ubuntu 20.04

--Insert the installation media, change the boot disk to "Each installation media" according to the BIOS of the motherboard, and boot to start the installation.

【important point】

--If you are adding an NVIDIA GPU, the following steps are required (I got stuck) --Select "Ubuntu (safe graphics)" from the boot menu to start "Ubuntu" (usually select "Ubuntu") --Turn on the "Graphics and ..." item in "Updates and Other Software" of the installer, and then continue the installation. --Since there is no screenshot, please check "Installer and Live Session" on the reference page.

[Ubuntu 20.04 Part 25-Known Issues with Ubuntu 20.04 LTS](https://kledgeb.blogspot.com/2020/04/ubuntu-2004-25-ubuntu-2004-lts.html)

--Confirmation of driver installation of nvidia --If the following message is displayed with the `` `$ nvidia-smi -l``` command, the nvidia driver is working.

python


$ nvidia-smi -l
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.100      Driver Version: 440.100      CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 166...  Off  | 00000000:01:00.0  On |                  N/A |
| 42%   37C    P8     8W / 125W |    249MiB /  5936MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1127      G   /usr/lib/xorg/Xorg                            35MiB |
|    0      2006      G   /usr/lib/xorg/Xorg                            94MiB |
|    0      2202      G   /usr/bin/gnome-shell                          97MiB |
|    0      6565      G   /usr/lib/firefox/firefox                       2MiB |
|    0      7875      G   /usr/lib/firefox/firefox                       2MiB |
|    0     10070      G   /usr/lib/firefox/firefox                       2MiB |
+-----------------------------------------------------------------------------+

3. Mount the internal HDD (I want to use SSD for OS and HDD for Data separately) </ summary>

[How to mount HDD of 2TB or more on CentOS](https://qiita.com/kubor/items/0a30c03a70a10c099df5)

--Find HDD

python


$ sudo parted -l

-Because it exceeds 2TB, attach a disc label with "GPT"

python


$ sudo parted /dev/sdb
(parted) mklabel gpt
(parted) p

--All capacity is ext4, 1 partition

python


(parted) mkpart primary ext4 0% 100%
(parted) p

--Since the partition is created, close parted.

python


(parted) quit

--Format with "ext4"

python


$ sudo mkfs.ext4 /dev/sdb1

--Create a mount point and mount the HDD (The mount point was just a directory / I wasn't sure if this location was good, so I did chmod777.)

python


$ sudo mkdir /home/Hdd_2nd
$ sudo mount -t ext4 /dev/sdb1 /home/Hdd_2nd
$ sudo mount

--Automount settings (write information to / etc / fstab)

python


$ sudo nano /etc/fstab

--Added the following line to the opened file ([tab], not Space between characters. I was addicted to it here)

python


/dev/sdb1       /home/Hdd_2nd   ext4    defaults        0       0

4. Mac-like basic settings and keyboard settings </ summary>

4.1. Keyboard settings: Compatible with US keyboards

--Try until it matches the keyboard (By the way, it is strange that the settings are different between [Apple Wireless Keyboard] and [MacBook Pro]. Even though the key layout is exactly the same)

python


$ sudo dpkg-reconfigure keyboard-configuration

--Write keyboard settings

python


$ sudo nano /usr/share/ibus/component/mozc.xml
<layout>us</layout>

4.2. Make the directory in your home English

――When accessing from the terminal, it is troublesome if it is Japanese, so return to English

python


$ LANG=C xdg-user-dirs-gtk-update

--If it doesn't work, edit `` `$ sudo nano ~ / .config / user-dirs.dirs``` as follows

python


$ sudo nano ~/.config/user-dirs.dirs
________________________
XDG_DESKTOP_DIR="$HOME/Desktop"
XDG_DOWNLOAD_DIR="$HOME/Downloads"
XDG_TEMPLATES_DIR="$HOME/Templates"
XDG_PUBLICSHARE_DIR="$HOME/Share"
XDG_DOCUMENTS_DIR="$HOME/Documents"
XDG_MUSIC_DIR="$HOME/Music"
XDG_PICTURES_DIR="$HOME/Pictures"
XDG_VIDEOS_DIR="$HOME/Videos"
________________________

--Rename the actual folder

python


$ mv $HOME/desktop$HOME/Desktop
$ mv $HOME/download$HOME/Downloads
$ mv $HOME/template$HOME/Templates
$ mv $HOME/Release$HOME/Share
$ mv $HOME/document$HOME/Documents
$ mv $HOME/musics$HOME/Music
$ mv $HOME/image$HOME/Pictures
$ mv $HOME/video$HOME/Videos

--It's OK to log in again (sometimes the sidebar display is mixed English / Japanese, but I gave up)

4.3. Match key bindings to Mac

4.3.1 Change "Left super (alt) key" and "Caps lock key" to "Ctrl key"

--Install Gnome Tweaks --Install from Ubuntu Software / [Tweaks] is the program name -[Tweaks]> Keyboard and mouse --Change the shortcut of the activity screen from "left super" to "right super" (to open the "left super key") -[Tweaks]> Keyboard and mouse> Additional layout options> Ctrl position> Select "Treat Caps Lock as Control" (change "Caps lock key" to "Ctrl key") -Please [Restart]

Tweak.png

4.3.2 Change "Left super key" to Ctrl key

--Keep a backup of your key assignments

python


$ xmodmap -pke > ~/.Xmodmap_default

--Find the key number of the "left super key" --When you type the following command, a square window will open, so if you "key in" while you can see it, the key number will be displayed (in my case, it was "133").

python


$ xev | grep keycode

キー入力.png Photoshop is the display when you press the "Return key" Change key binding Basic setting method of Xmodmap

--Add the following to the opened file

python


$ nano ~/.Xmodmap
________________________
clear control
clear mod4

! Command_L -> Control_L
keycode 133 = Control_L

add control = Control_L Control_R
add mod4 = Super_L Super_R Super_L Hyper_L
_________________________

--Reflect the settings

python


$ xmodmap ~/.Xmodmap

--If you make a mistake, restore the backup file

python


$ xmodmap ~/.Xmodmap_default

-Please [Restart]

** The Ctrl key on the left half of the keyboard has a gorgeous keyboard layout of "3 (left Super (alt) / Caps Lock / original ctrl key)" ** ** Now you can copy with "Left Super (alt) key" + "c" **

4.3.3 Convert to "half-width English" with "Option (Start) key" + "s"

--mozc settings: GUI tool installation --By default, the "mozc" GUI tool is not installed, so install it.

python


$ sudo apt install mozc-utils-gui

--Settings > Region and Language > Input Source > Mozc Gear Icon > Select Key Settings > Select Kotoeri

** With this alone, "Option (Start) key" + "s / a / c / x / z" will give you a familiar feeling of "half-width katakana / half-width English / hiragana / full-width katakana / full-width English". Mozc, Elai **

4.3.4. English / Japanese input source switching with "Left Super (alt) key" + "Space key" (for US keyboard)

--Change input source switching shortcut key (US keyboard compatible) --Settings > Keyboard shortcuts > Set both "Switch to previous input source" and "Switch to next input source" to [Disable] in the typing window.

--Mozc property window > Select key setting > Edit > Mozc key setting screen > "Hankaku / Zenkaku" input key (4 places) > Change to [Ctrl + Space]

** Now you can switch between English and Japanese with "Left Super (alt) key" + "Space key" **

5. Installation of basic software </ summary>

[What to do after installing Ubuntu 20.04](https://ohmyenter.com/things-to-do-after-ubuntu-20-04-installation/)

5.1. Uninstall unnecessary software

--Ubuntu Software > Installed > "Delete" unnecessary software

5.2. CopyQ installation (copy / paste enhancement software): If necessary

python


$ sudo apt install copyq
  • basic configuration --Starts automatically Turn on --Global Shortcut Show main window under mouse cursor Alt + Ctrl + V

5.3. Dump installation: Backup software (for full system backup): If necessary

python


$ sudo apt install dump

--Actual backup command (I tried backing up to a data HDD)

python


$ sudo dump -0 -u -j /dev/sda2 -f /home/hdd_2nd/Sys_Backup/backup_2020_08_08.dump

5.4. Google Chrome installation (Google Translate browser): If necessary

Download and save 64bit .deb (for Debian / Ubuntu) from Official Site

python


$ cd Downloads
$ sudo apt install -y ./google-chrome-stable_current_amd64.deb
[How to install Google Chrome on Ubuntu](https://linuxfan.info/google-chrome-on-ubuntu)

5.5. After all, I want to feel free to use DeepP Translator on Ubuntu: If necessary

[DeepL Translator on Linux](https://qiita.com/masan4444/items/030080c74b53a771c132) --There is a description that Script should be written with the full path, but the full path of this software is "sh /home/username/deepl-clip/app.sh".

5.6. Thunderbird settings

--You need to get a "special password" from the Apple ID homepage to set up an iCloud email account. --Other than that, it's normal

5.7. Enhanced Mozc dictionary

[Kana-English / Japanese-English conversion with Mozc (Google IME)](http://blog.blueblack.net/item_442)

--Refer to the above, install "Kana-English dictionary v2.txt" and "Japanese-English dictionary.txt" in "New dictionary" with "Mozc-> Dictionary tool" (It is troublesome to import to the selected dictionary by mistake. Please note) --If you are concerned about the priority of dictionaries, please refer to "Avoiding priority of user dictionaries" on the reference page.

5.8. Install virus software (for communication with the main PC)

--Install "ClamTk" with "Ubuntu Software"

5.9. Data exchange with the main PC

――It seems that there are various methods, but the following services that exchange via a browser that does not require system modification are safe. - OneDrive - GoogleDrive

6. Installation of basic development software </ summary>

6.1 Shell manual content in Japanese: Those who are not good at English

--Confirm that the environment variable LANG of the shell is "ja_JP.UTF-8"

python


$ echo $LANG

--If not, change

python


$ export LANG=ja_JP.utf8

--Reconfirm the changed contents

python


$ echo $LANG

--Install the Japanese manual

python


$ sudo apt-get install manpages-ja

6.2 fish shell and fisher installation: only for those who like

――I put in a shell that has a reputation for being easy to use.

python


$ sudo apt-add-repository ppa:fish-shell/release-3
$ sudo apt-get update
$ sudo apt-get install -y fish

$ sudo apt install curl /Not required if curl is already installed
$ curl https://git.io/fisher --create-dirs -sLo ~/.config/fish/functions/fisher.fish
$ fisher add oh-my-fish/theme-bobthefish

** ** **-Do not change the login shell from bash **

--You have not set fish as your login shell. Dedicated to terminator.

6.3 terminator installation: only for those who like

--You can open multiple shells with ROS, so put them in.

python


$ sudo apt install terminator

--Start and right-click in the terminal to "Set": The following is your choice --Default profile background is now 0.8 transparent --In the command of the default profile, check [Execute custom command instead of login shell] and enter "/ usr / bin / fish". --Change the font to one of "○○○ for Powerline" in "Profile> General> Font" (after installing the font in 6.4)

6.4 Install Powerline fonts: Only for those who like

--A font that is essential for making a cool terminal (it seems to be easy to see) --Set only for fish shell

python


$ git clone https://github.com/powerline/fonts.git --depth=1
$ cd fonts
$ ./install.sh
$ cd ..
$ rm -rf fonts

6.5. Git install: only those who need it

――It is essential for development (especially ROS), but let's put it in when needed

python


$ sudo apt install git

--Set the minimum settings.

python


$ git config --global user.name "username"
$ git config --global user.email email address
$ git config --global core.editor nvim

--Public key registration on github --Key authentication is convenient when using the git protocol.

python


$ sudo apt install xclip "Not required if installed"
$ bash
$ ssh-keygen -t rsa -b 4096 -C "mail address"
~ If you don't use the passphrase, enter repeatedly ~
$ eval "$(ssh-agent -s)"
$ ssh-add ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub | xclip -selection c "Copy to clipboard"
~ Register SSH KEY from Settings page on github ~
Paste the public key copied to the clipboard with the xclip command

6.6. Install Visual Studio Code: Recommended

--Defactostandard (?) Of Program Editor on Ubuntu ――Since there are many functions, let's unify the Editor of Windows at work and Mac at home to VS Code to get used to it.

-Download and install the Linux version from the Download page

  • Configuration -Is it in your PATH? - code -v --OK when the version comes out --If you don't have PATH -Enter [Command] + [Shift] + [P] keys to display the "Command Palette". ・ Shell Command: Install'code' command in PATH)

--Easy to use --Click Extension on the left side of the screen to open MARKET PLACE --Install Remote Development: For Docker --Install Docker: For Docker ――Japanese - -・ How to translate into Japanese with Visual Studio Code Windows --Restart Visual Studio Code just in case

6.7. Now that we have a break, back up the system: only those who need it

--Back up

python


$ sudo dump -0 -u -j /dev/sda2 -f /home/inata/hdd_2nd/Sys_Backup/backup_2020_08_08.dump

(For the backup source, enter up to the number at the end (2 of / dev / sda2))

7. Install Docker (for NVIDIA) </ summary>

[How to build a deep learning GPU learning environment with Docker](https://qiita.com/karaage0703/items/e79a8ad2f57abc6872aa)

7.1. Docker-GPU settings

7.1.1 Disable Secure Boot

--It seems that the nvidia-smi command does not work well on Docker, so be sure to execute it. --In my environment, the nvidia-smi command worked, but I'm worried, so I "disable" it properly. --Change in BIOS (ASUS motherboard was disabled by "Delete Secure Boot Key")

7.1.2 Confirmation of nouveau stop

--It should be stopped when setting the GPU at the time of installation, so check it

python


$ lsmod | grep -i nouveau

--OK if nothing is displayed

7.1.3 NVIDIA driver updates

--Software and Updates> Select the (proprietary, verified) driver from the additional drivers and [Restart]. 追加のドライバー.png ** [Trouble] ** If you select and install the (proprietary, verified) driver in "Additional drivers",

python


$ nvidia-smi
$ Failed to initialize NVML: Driver/library version mismatch

(Originally, it is correct to display the GPU status of 7.5.)

The situation does not change even if you enter commands or install

python


$ sudo ubuntu-drivers autoinstall

-(Proprietary, no verification) driver will correctly give the result of the `` `$ nvidia-smi``` command (probably), so leave it as it is for a while

――There is a newer driver on the NVIDIA homepage, but it seems that there are many cases where you will be addicted to it by yourself, so for the time being, Ubuntu's "Recommended for the second" driver is [good].

7.2. Installing nvidia-cuda-toolkit

--Software required when you want to use an NVIDIA GPU card for general-purpose numerical calculation purposes. - “If you use NVIDIA Docker, you don't need to include CUDA or cuDNN as long as you install the NVIDIA driver. You can use any image you like as long as the NVIDIA driver supports it.” ① </ font>

python


$ sudo apt install nvidia-cuda-toolkit

7.3. Docker installation

――The procedure of the official version is complicated, and I also messed up once, so in [Karaage-san], How to build a deep learning GPU learning environment with Docker I will use the script described (Karaage-san, thank you)

python


$ sudo apt install -y curl (not required if installed)
$ curl -s https://raw.githubusercontent.com/karaage0703/ubuntu-setup/master/install-docker.sh | /bin/bash

-Please [Restart] --Confirmation of Docker installation

python


$ sudo docker run hello-world
OK if there are no errors

--Clean up

python


$ docker container prune
$ docker image rm hello-world:latest

7.4. Installing the NVIDIA Container Toolkit

--Software for using the GPU on the host from the Docker container ――I will also use the script of "Karaage-san" (Karaage-san, thank you again)

python


$ curl -s https://raw.githubusercontent.com/karaage0703/ubuntu-setup/master/install-nvidia-container-toolkit.sh | /bin/bash

--Startup confirmation

python


$ docker run --gpus all --rm nvidia/cuda nvidia-smi

――I get an error for some reason, but ignore it (Karaage-san also ignored it (laughs))

7.5. Checking the operation of NVIDIA GPU on Docker

--Run the official TensorFlow Docker image

python


$ docker run --gpus all -it --rm --name tensorflow-gpu -p 8888:8888 tensorflow/tensorflow:latest-gpu-py3
(-p 8888:8888 is for the port of jupyter notebook)

--If the following message is displayed with the $ nvidia-smi command, the GPU is running on Docker.

python


# nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.100      Driver Version: 440.100      CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:1Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 166...  Off  | 00000000:01:00.0  On |                  N/A |
| 42%   36C    P8     8W / 125W |    342MiB /  5936MiB |      2%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

【Note】 **-The process details are not displayed even if the `` `$ nvidia-smi``` command is executed on Docker. This seems to be a specification. ** ** ** ・ If you ask a question on StackOverflow, [You will not be able to see, due to driver not being aware of PID namespace. (DeepL Translator: You cannot see it because the driver does not recognize the PID namespace.)] Was answered **

--You can access Jupyter Notebook from this state

python


# jupyter notebook

--jupyter notebook starts and the address is displayed on the terminal

python


http://127.0.0.1:8888/?token=xxxxxxxxxxxxxx`

--Right-click the address and select "Open Address" to open the browser and launch "Jupyter Notebook". --When the GPU is displayed by executing the following command, the GPU is recognized by "Jupyter Notebook" on Docker.

python


from tensorflow.python.client import device_lib
device_lib.list_local_devices()

Jupyter Notebook.png

--When you can confirm, close Jupyter Notebook. --Exit Docker

python


# exit

7.6. Delete the Docker container and image that I ran for trial

python


$ docker container prune
$ docker image rm tensorflow/tensorflow:latest-gpu-py3

7.10. Studying Docker: Only for those who need it

――It's easier to know the mechanism and commands of Docker later, so let's study a little around here. --Since the installation of Docker is finished, it is OK if you learn after that. --I bought a Udemy course on sale (example) -Docker starting from Linux command-BE A FIRST PENGUIN AND GROW AS WHALE ――It was good that this course, "Volume that ends in one day", "PowerPoint materials can be obtained in PDF", and "Understanding Docker construction on a virtual environment of Mac or Windows"

――Because it is a good idea, save the Dockerhub account as well.

8. Build [TensorFlow + Keras] environment on Docker </ summary>

--Use the NGC TensorFlow image provided by NVIDA. I don't know the difference from the official TensorFlow Docker image (I haven't checked it), but I came here to use NVIDIA's GPU, so it's [If you eat poison, it's a plate].

8.1. Installing cuDNN

--I don't know if CuDNN is also needed, but I need a version to determine the image of NGC, so I will include it. - “If you use NVIDIA Docker, you don't need to include CUDA or cuDNN as long as you install the NVIDIA driver. You can use any image you like as long as the NVIDIA driver supports it.” ① </ font>

--How to check the version of CUDA

python


$ nvcc -V

--Go to NVIDIA cuDNN Home Page --Register as a user and download the deb file corresponding to the installed CUDA version. --Downloaded runtime and Developer Library - cuDNN Runtime Library for Ubuntu18.04 (Deb) - cuDNN Developer Library for Ubuntu18.04 (Deb)

--Execute the installation with the dpkg command (the order of installation seems to be important)

python


$ sudo dpkg -i libcudnn7_7.6.5.32-1+cuda10.1_amd64.deb
$ sudo dpkg -i libcudnn8-dev_8.0.2.39-1+cuda10.1_amd64.deb
$ sudo dpkg -i libcudnn8-doc_8.0.2.39-1+cuda10.1_amd64.deb

8.2. Check NVIDIA driver version

--How to check the version of CUDA

python


$ nvcc -V

--How to check the version of CuDNN

python


$ cat /usr/include/cudnn.h | grep CUDNN_MAJOR -A 2
#define CUDNN_MAJOR 7
#define CUDNN_MINOR 6
#define CUDNN_PATCHLEVEL 5
--
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

-** In this case, it will be "7.6.5" **

8.3. Select NGC image for Docker for TensorFlow

――Determine the image to PULL from the following homepage and CUDA and CuDNN versions. - https://ngc.nvidia.com/catalog/containers/nvidia:tensorflow/tags - https://docs.nvidia.com/deeplearning/frameworks/tensorflow-release-notes/ --In my case, ** CUDA: 10.289 / cuDNN: 7.6.5, so I PULL (RUN) "TensorFlow Release 20.03" **

8.4. Get NGC image of Docker for TensorFlow and start container

--Get NGC image of Docker for TensorFlow

python


$ docker image pull nvcr.io/nvidia/tensorflow:20.03-tf1-py3

--Launch the NGC image as a container with the name "tensor_keras"

python


$ docker run --name tensor_keras --gpus all -it -p 8888:8888 nvcr.io/nvidia/tensorflow:20.03-tf1-py3

8.5. Check the internal files of the container and install the missing parts

--Check for already installed programs

python


# pip list

――It's installed quite a bit, so I'll install it as needed. --I put the following for the time being

python


# pip install --upgrade pip
# pip install Keras
# pip install pandas
# pip install pandas_datareader

--Refer to the following chapter, launch "Jupyter Notebook" and check the operation of "GPU" in the same way. --## 7.5. Checking the operation of NVIDIA GPU on Docker

8.6. Save the modified container contents to a Docker image file

――Originally, managing with a Docker file would be cooler without consuming disk space, but since it is a container installed by PULL, create a new Docker image from the modified container.

8.6.1. Stop the Docker container you want to save

python


# exit

8.6.2. Make sure the target container is stopped

――It is OK if "STATUS" is "Exited"

python


$ docker container ls -a
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                     PORTS               NAMES
70d0795d84e8        inata/my_tensorflow_img:1.0   "/usr/local/bin/nvid…"   3 weeks ago         Exited (0) 4 seconds ago                       tensor_keras

8.6.3. Save to image file

--Save the modified container as an image file named "my_tensorflow_img: 1.0" --Here, if you consider uploading to Docker Hub, you will need the Docker Hub account name (inata).

python


$ docker container commit -a "INA-Ta" tensor_keras inata/my_tensorflow_img:1.0

--If you only save locally, you can pass without the signature "-a" INA-Ta "" or "account name"

python


$ docker container commit tensor_keras my_tensorflow_img:1.0

8.6.4. Container driven from modified Docker image

--If you want to start a new container in the future, you can start from the changed state by starting from the saved image file (it is a save point). --Launch a container with the name "tensor_kera" for the "my_tensorflow_img: 1.0" image file.

python


$ docker run --name tensor_keras --gpus all -it -p 8888:8888 inata/my_tensorflow_img:1.0

8.7. Save the modified image file to Docker hub

--Log in to Docker Hub

python


$ docker login

--Push the newly created image file to Docker Hub

python


$ docker image push inata/my_tensorflow_img:1.0

DockerHub.png

--When you're done, log out

python


docker logout

8.8. Save / read the changed image file in the local folder

--The Dockerhub file also says "Delete the container image that has not been used for 6 months", so save it in the local file as well. The save destination is the HDD for data. Saved as a tar file

python


$ docker image save -o /home/Hdd_2nd/Docker_Image/my_tensorflow_img_1.0.tar inata/my_tensorflow_img:1.0

--Read an image file from a local file

python


$ docker image load -i /home/Hdd_2nd/Docker_Image/my_tensorflow_img_1.0.tar

** [Additional update ongoing] m (_ _) m **

9. Build [ROS2] environment on Docker [Continue to update] </ summary>

** [Additional update ongoing] m (_ _) m **

10. Build [PyTorch] environment on Docker [Update ongoing] </ summary>

** [Additional update ongoing] m (_ _) m **

change history

Version 0: September 21, 2020

1st edition: September 22, 2020 ① </ font>

- Reflects comments from Karaage-san (Thank you: Karaage-san) </ font>

2nd Edition: September 25, 2020

--Since it is too long, change to the foldable display