From nothing on Ubuntu 18.04 to setting up a Deep Learning environment in Tensor

Introduction

Apart from my research, I decided to touch CNN as a part-time job, so I thought I couldn't run the GPU on my mac and couldn't easily turn it, so I'm going to run Linux, which I hadn't been able to handle! This is an article I wrote for the time being.

To put it simply, I got a user from a Linux server in the laboratory, so it is a reminder when I was able to run the GPU with Tensor etc. Long live your memo.

Basically, we will proceed while referring to this Running TensorFlow-gpu with ubuntu 18.04.

English but [A step by Step Guide to Install Tensorflow GPU on Ubuntu 18.04 LTS](https://medium.com/@kekayan/step-by-step-guide-to-install-tensorflow-gpu-on-ubuntu-18 -04-lts-6feceb0df5c0) was also helpful.

environment

procedure

Change Password

Now that you have the user with sudo privileges, let's change the password

$ passwd [username]

(current) UNIX password: 
Enter new UNIX password: 
Retype new UNIX password: 

Enter your current password and then enter your new password twice to complete.

passwd: password updated successfully

Nvidia driver installation

For the time being, check the current status with the following command.

$ nvidia-smi

Then,

Wed Nov 20 13:44:59 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.129                Driver Version: 390.129                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K620         Off  | 00000000:02:00.0  On |                  N/A |
| 34%   39C    P8     1W /  30W |    124MiB /  1994MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  Off  | 00000000:03:00.0 Off |                  N/A |
| 23%   40C    P8    26W / 250W |      2MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2599      G   /usr/lib/xorg/Xorg                            49MiB |
|    0      2929      G   /usr/bin/gnome-shell                          71MiB |
+-----------------------------------------------------------------------------+

It seems that it is already installed. So next, I will introduce CUDA cuDNN.

Installation of CUDA 9.0 and cuDNN

This is also confirmed by the following command for the time being.

$ nvcc -V
Command 'nvcc' not found, but can be installed with:

apt install nvidia-cuda-toolkit
Please ask your administrator.

It seems that this needs to be installed. It says Please ask your administrator, I'm glad I got the administrator authority.

Let's proceed with the installation.

This is for reference Run TensorFlow-gpu on ubuntu 18.04 Or Put Tensorflow with GPU in Ubuntu 18.04 Then, the latest CUDA 10.0 says that Tensorflow-gpu is not supported, so CIDA 9.0 was installed.

Both articles are about a year ago, so as of November 20, 2019, if you look at the Tensorflow official website,

The following NVIDIA® software must be installed on your system: NVIDIA® GPU drivers —CUDA 10.0 requires 410.x or higher. CUDA® Toolkit —TensorFlow supports CUDA 10.0 (TensorFlow >= 1.13.0) CUPTI ships with the CUDA Toolkit. cuDNN SDK (>= 7.4.1) (Optional) TensorRT 5.0 to improve latency and throughput for inference on some models.

Was written.

CUDA® Toolkit —TensorFlow supports CUDA 10.0 (TensorFlow >= 1.13.0) So, it seems that CUDA 10.0 is okay if you use TensorFlow with 1.13.0 or higher.

However, it says NVIDIA® GPU drivers — CUDA 10.0 requires 410.x or higher.. What version of nvidia was that? I will return to confirm.

$ nvidia-smi
Wed Nov 20 13:44:59 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.129                Driver Version: 390.129                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K620         Off  | 00000000:02:00.0  On |                  N/A |
| 34%   39C    P8     1W /  30W |    124MiB /  1994MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  Off  | 00000000:03:00.0 Off |                  N/A |
| 23%   40C    P8    26W / 250W |      2MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2599      G   /usr/lib/xorg/Xorg                            49MiB |
|    0      2929      G   /usr/bin/gnome-shell                          71MiB |
+-----------------------------------------------------------------------------+

It was 390.120. It seems to either update nvidia or set CUDA to 9.0.

Is it okay for the GPU I'm using in the first place? I thought, and confirmed it at here.

スクリーンショット 2019-11-20 14.33.03.png Enter the product name and click SEARCH. スクリーンショット 2019-11-20 14.33.16.png

It was recommended to install the 440.31 version of the driver. Apparently it's okay at all.

After updating nvidia and installing TensorFlow 1.13.0 and CUDA 10.0, or worrying about TensorFlow 1.12.0 or earlier and CUDA 9.0 for about 30 seconds, I decided to go with the latter.

Well, it's a shared server, so it would be a problem if there was an update, and it's safer to be one before the latest.

By the way, in order to investigate the version relationship of Nvidia, CUDA, TensorFlow, etc., this article → [Nvidia driver, CUDA, cuDNN, Tensorflow-gpu and Python version correspondence](https://qiita.com/konzo_ / items / a6f2e8818e5e8fcdb896) Was very helpful! Thank you.

Finally, proceed to install CUDA 9.0. Access the official NVIDIA installation page → CUDA Toolkit 9.0 Downloads Select to. Is it okay because there is no 18.04? I thought, but when I looked it up, it seems that 17.04 is okay.

スクリーンショット 2019-11-20 14.57.10.png スクリーンショット 2019-11-20 16.56.10.png

Since I am working locally this time, I will download it and send it to the server with scp.

local


$ scp -P [port number] Downloads/cuda-repo-ubuntu1704_9.0.176-1_amd64.deb [username]@[IP address]:[Copy destination path]

Normally port specification -p, but when scp, it is -P.

$ wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/[deb file]

But it's okay.

After that, proceed with the installation according to the official website. ** In addition, it seems that the latest version will be installed with the last sudo apt-get install cuda, so it seems important to specify the version corresponding to the end. This time, -toolkit-9-0 is added. ** **

$ sudo dpkg -i cuda-repo-ubuntu1704_9.0.176-1_amd64.deb
$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1704/x86_64/7fa2af80.pub
$ sudo apt-get update
$ sudo apt-get install cuda-toolkit-9-0

Next, it seems that work to pass through the PATH is necessary. Enter the following command and edit ".bashrc".

$ nano ~/.bashrc

Enter the following character string at the end and save.

.bashrc


export PATH=/usr/local/cuda-9.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Reboot

$ sudo reboot

After starting, check if CUDA is installed with the following command.

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

Is displayed, so OK.

cuDNN installation

cuDNN is a multi-library installed with CUDA, and seems to be for high-speed calculation on GPU using CUDA.

You need to access the following site and register as a member. https://developer.nvidia.com/rdp/cudnn-download

After that, install the following 3 from Download cuDNN v7.6.4 for CUDA 9.0.

Execute the following commands in order in the downloaded directory. Change the version to the downloaded version as appropriate.

$ sudo dpkg -i libcudnn7_7.6.4.38-1+cuda9.0_amd64.deb
$ sudo dpkg -i libcudnn7-dev_7.6.4.38-1+cuda9.0_amd64.deb
$ sudo dpkg -i libcudnn7-doc_7.6.4.38-1+cuda9.0_amd64.deb

This completes the installation of cuDNN.

Installation of pyenv and Anaconda, environment construction (Python environment construction)

I want to create an environment where I can install Anaconda from pyenv and create a virtual environment with Anaconda while making it possible to switch the version of Anaconda with pyenv at any time.

pyenv

$ git clone https://github.com/yyuu/pyenv.git ~/.pyenv
$ nano ~/.bashrc

After installing, add the following to .bashrc.

.bashrc


export PYENV_ROOT=$HOME/.pyenv
export PATH=$PYENV_ROOT/bin:$PATH
eval "$(pyenv init -)"

After rebooting and confirming that the pyenv path is in place, proceed to install Anaconda.

Anaconda

$ pyenv install --list | grep anaconda

Check the version of Anaconda that can be installed with. This time, install 5.2.0.

$ pyenv install anaconda3-5.2.0

Switch the version and check if it is reflected.

$ pyenv global anaconda3-5.2.0
$ pyenv versions
  system
* anaconda3-5.2.0 (set by /home/okuda/.pyenv/version)
$ python --version
Python 3.6.5 :: Anaconda, Inc.

Looks good.

Next, create a virtual environment with anaconda.

$ conda -V
conda 4.5.4

Make sure it's working properly

$ conda create -n tensor pip python=3.6

Create a virtual environment named tensor. Install pip and python3.6 together.

And I would like to switch the virtual environment, When using anaconda on the pyenv environment, conda activate cannot be used as it is.

$ conda activate tensor
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
If your shell is Bash or a Bourne variant, enable conda for the current user with

    $ echo ". /home/okuda/.pyenv/versions/anaconda3-5.2.0/etc/profile.d/conda.sh" >> ~/.bashrc

or, for all users, enable conda with

    $ sudo ln -s /home/okuda/.pyenv/versions/anaconda3-5.2.0/etc/profile.d/conda.sh /etc/profile.d/conda.sh

The options above will permanently enable the 'conda' command, but they do NOT
put conda's base (root) environment on PATH.  To do so, run

    $ conda activate

in your terminal, or to put the base environment on PATH permanently, run

    $ echo "conda activate" >> ~/.bashrc

Previous to conda 4.4, the recommended way to activate conda was to modify PATH in
your ~/.bashrc file.  You should manually remove the line that looks like

    export PATH="/home/okuda/.pyenv/versions/anaconda3-5.2.0/bin:$PATH"

^^^ The above line should NO LONGER be in your ~/.bashrc file! ^^^

I get angry.

If your shell is Bash or a Bourne variant, enable conda for the current user with

$ echo ". /home/okuda/.pyenv/versions/anaconda3-5.2.0/etc/profile.d/conda.sh" >> ~/.bashrc

Because it is written as it is

$ echo ". /home/okuda/.pyenv/versions/anaconda3-5.2.0/etc/profile.d/conda.sh" >> ~/.bashrc

If you run and read bashrc again

$ source .bashrc
$ conda activate tensor
(tensor) :~$ 

It switched safely!

The rest is the installation of the necessary libraries. First, install the famous framework TensorFlow.

TensorFlow-GPU installation

The version is on the TensorFlow official website スクリーンショット 2019-11-20 18.53.45.png スクリーンショット 2019-11-20 18.52.49.png So, this time, download tensorflow_gpu-1.12.0.

$ pip install tensorflow-gpu==1.12.0

When it is completed, check the operation to see if TensorFlow works on the GPU.

$ python 
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>>import tensorflow as tf
>>> tf.test.gpu_device_name()
2019-11-20 19:02:23.713902: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-20 19:02:24.115406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:03:00.0
totalMemory: 10.92GiB freeMemory: 10.76GiB
2019-11-20 19:02:24.278457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties: 
name: Quadro K620 major: 5 minor: 0 memoryClockRate(GHz): 1.124
pciBusID: 0000:02:00.0
totalMemory: 1.95GiB freeMemory: 1.80GiB
2019-11-20 19:02:24.278929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1496] Ignoring visible gpu device (device: 1, name: Quadro K620, pci bus id: 0000:02:00.0, compute capability: 5.0) with Cuda multiprocessor count: 3. The minimum required count is 8. You can adjust this requirement with the env var TF_MIN_GPU_MULTIPROCESSOR_COUNT.
2019-11-20 19:02:24.278972: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-11-20 19:02:24.784359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-11-20 19:02:24.784436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1 
2019-11-20 19:02:24.784450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N N 
2019-11-20 19:02:24.784459: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   N N 
2019-11-20 19:02:24.784905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 10403 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1)
'/device:GPU:0'

Do you recognize another GPU? I will investigate it a little later. It's OK because it became'/ device: GPU: 0'.

Install Jupyter Lab

From here, I personally like Jupyter, so I will add it. I like it.

$ conda install jupyterlab

After that, do what you did with here so that you can access it remotely, and you're done.

Surprisingly, it is difficult to use GPU. Thank you for your hard work.

Recommended Posts

From nothing on Ubuntu 18.04 to setting up a Deep Learning environment in Tensor
From setting up a Rust environment to running Hello World
Build a "Deep learning from scratch" learning environment on Cloud9 (jupyter miniconda python3)
From setting up Raspberry Pi to installing Python environment
Setting up Jupyter Lab in a Python 3.9 venv environment
Steps to quickly create a deep learning environment on Mac with TensorFlow and OpenCV
How to build a new python virtual environment on Ubuntu
A note on how to load a virtual environment in PyCharm
Building a Python environment on Ubuntu
From Ubuntu 20.04 introduction to environment construction
Deep learning tutorial from environment construction
How to build a Python environment using Virtualenv on Ubuntu 18.04 LTS
From installing Ansible to building a Python environment in Vagrant's virtual environment
Shell script to build pyenv environment on ubuntu in one shot
Everything from building a Python environment to running it on Windows
[Deep Learning from scratch] About the layers required to implement backpropagation processing in a neural network
Reinforcement learning to learn from zero to deep
How to install Deep Learning framework Caffe on Mac in CPU mode
Steps to install Python environment on Ubuntu
A story of a deep learning beginner trying to classify guitars on CNN
Image alignment: from SIFT to deep learning
"Deep Learning from scratch" in Haskell (unfinished)
[Windows 10] "Deep Learning from scratch" environment construction
How to read environment variables from .env file in PyCharm (on Mac)
Introduction to docker Create ubuntu environment in ubuntu
Deep Learning from scratch-Chapter 4 tips on deep learning theory and implementation learned in Python
Use dHash to locate on the course from a scene in a racing game
Build a python environment to learn the theory and implementation of deep learning
I tried to extract a line art from an image with Deep Learning
Create a setting in terraform to send a message from AWS Lambda Python3.8 to Slack
Build a PYNQ environment on Ultra96 V2 and log in to Jupyter Notebook
How to install the deep learning framework Tensorflow 1.0 in the Anaconda environment of Windows
Ubuntu18.04.05 Creating a python virtual environment in LTS
Deep Learning from scratch ① Chapter 6 "Techniques related to learning"
Good book "Deep Learning from scratch" on GitHub
[Latest] How to build Java environment on Ubuntu
Install python package in personal environment on Ubuntu
Notes on machine learning (updated from time to time)
[Venv] Create a python virtual environment on Ubuntu
Build a Kubernetes environment for development on Ubuntu
Set up a Python development environment on Marvericks
How to build Java environment on Ubuntu (Linux)
How to set up a Google Colab environment with Coursera's advanced machine learning courses
Build a machine learning environment using PyCharm on Ubuntu environment (TensorFlow will also be introduced!)
I tried to build an environment with WSL + Ubuntu + VS Code in a Windows environment
Try to make a blackjack strategy by reinforcement learning (② Register the environment in gym)
On a PC that cannot boot from NVMe, move / usr etc. to NVMe to speed up
How to create a Python 3.6.0 environment by putting pyenv on Amazon Linux and Ubuntu
Memo A beginner tried to build a Java environment and Japaneseize it on Ubuntu 18.04.2 LTS.
Operate mongoDB from python in ubuntu environment ① Introduction of mongoDB
How to set up a Python environment using pyenv
# 2 Build a Python environment on AWS EC2 instance (ubuntu18.04)
Set up a file server on Ubuntu 20.04 using Samba
Build a machine learning Python environment on Mac OS
A memorandum on how to use keras.preprocessing.image in Keras
I made a Python3 environment on Ubuntu with direnv.
How to build a Django (python) environment on docker
Set up a free server on AWS in 30 minutes
Learning Deep Forest, a new learning device comparable to DNN
[Deep Learning from scratch] I tried to explain Dropout
How to build a Python environment on amazon linux 2