[DOCKER] Write about building a Python environment for writing Qiita Qiita

Let's write a Qiita article on machine learning for the first time in a while! !! !! I thought, but every time it was troublesome to build the environment, it was a hassle, so it was a meta-like thing that I tried to build an environment for it? I would like to write an article.

Overview

In this article, I would like to write about how to easily set up an independent python environment and connect from a browser using Docker-compose. We'll also write about how to execute code interactively from VS Code as well as access from a browser! Even if these are not for writing Qiita, I think that they can answer requests such as "I want to run it in a clean execution environment"!

Premise

This article does not cover the introduction of Docker, Docker-compose and VS Code. If you look it up, it will come out easily (Docker on Windows seems to be difficult, but ...). Also, since I specialize in data science, the programming language is Python3. I won't write in detail, but R is also partially supported. Since I am an Apple believer, I use a Mac as the OS, but I think that it will probably work in any environment as long as Docker can be used.

procedure

In this article, we will take the following steps:

  1. Download Dockerfile
  2. Create docker-compose.yml
  3. Start the container from docker-compose and access from the browser
  4. (Bonus) Comfortable coding with VS Code's Remote Desktop and jupyter's Extension

1. Download Dockerfile

The docker-stacs repository on jupyter official github has many tools for building a python environment using Jupyter. For example, if you want to do Deep Learning with tensorflow, go to tensorflow-notebook and it will look like below. You can see that there is a Dockerfile in. image.png

In this Dockerfile

ARG BASE_CONTAINER=jupyter/scipy-notebook

There is a part called, but this is said to be an additional operation based on scipy-notebook It makes sense. And if you look at scipy-notebook, it's the same.

ARG BASE_CONTAINER=jupyter/minimal-notebook

It is written that. This is because the environment for tensorflow is the environment for Scipy (for data science with basic python) and tensorflow is added, and the environment for Scipy is the environment where scikit-learn and pandas are added to the minimal environment. It means that there is. I won't go into details, but there are other spark environments and R environments, and if you use the Dockerfile as needed, the environment will be launched in one shot! Also, since it is Docker, if you have a library you want to add personally, you only need to add a little, which is very convenient!

So, first open the directory you want to run from the terminal and clone this repository.

$ git clone https://github.com/jupyter/docker-stacks.git

And, for example, when you want to use base-notebook,

$ cp -R ./docker-stacks/base-notebook ./

If you copy the tool, the download is complete.

2. Create docker-compose.yml

It is troublesome to start jupyter every time and write a command that specifies the port, so I will start it easily using docker-compose.

docker-compose.yml


version: '3'
services:
  #Service name jupyterlab(Container name)Set to
  jupyterlab:
    #Build from Dockerfile(To be able to write additional settings)
    build:
      context: .
      dockerfile: Dockerfile

    #host's./work in the container/home/Mount on jovyan
    # Jovian means "related to Jupiter"
    volumes:
       - "./work:/home/jovyan"
      
    #Set user
    user: root
    environment:
      B_UID: 1000
      NB_GID: 100
      GRANT_SUDO: "yes"

    #Allow access to the device
    privileged: true

    #Map host 8888 to container 8888
    ports:
      - "8888:8888"

    #Reboot unless stopped
    restart: unless-stopped

    #Start jupyterlab after starting the container
    command: start.sh jupyter lab --NotebookApp.token=''

It is OK if you create the above file in the same hierarchy as the Dockerfile (I am directly under the base-notebook directory). However, if port 8888 is in use by the host, it is a good idea to set XXXX: 8888 to a different number.

3. Start the container from docker-compose and access from the browser

After doing the above, go to the same level as docker-compose.yml in the terminal,

$ docker-compose up -d

Just hit. Then, when it starts up, type localhost: 8888 in your browser![Image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/265221/f45f76e0-4847 -8d03-4309-ddb2a79e5cd6.png) The jupyter lab is opened like this. It's really easy. Since the volume is set to ./work /home/jovyan, if you create a file from the home of jupyterlab, it will be reflected in ./work on the host side. You have now built a python environment. All you have to do is install the required packages according to the article you want to write, and share the Dockerfile for reproducibility!

4. (Bonus) Comfortable coding with VS Code's Remote Desktop and jupyter's Extension

This is enough, but this time I will challenge to create an interactive execution environment with VS Code as a bonus. The advantage of doing this is that you can still partially execute the py file. I think it's very convenient to be able to easily debug and write processes that are difficult only for ipynb files, such as refactoring.

Then, I will start work immediately. If VS Code with Docker and Remote Desktop Extension installed, press Command + Shift + P to open the command operation tab as shown below. image.png Use Open Folder in Container to open the tool you just set. (In my case, open the base-notebook) image.png

Now, when asked for Config, choose From docker-compose. Then wait for a while, image.png If such a screen is opened, access to the container is successful for the time being.

Speaking of jupyter, type jupyter from the Extension search and install this one displayed at the top. image.png

When the installation is complete, the button "Reload required" will appear as shown below, so press it to reload. image.png

After reloading, let's create a python file by deciding appropriately as jupyter_test.py in ./work. Then type "# %%" as shown below and you'll see commands like "Run Cell" above it! image.png When you press this Run Cell, the code written in the block up to the next "# %%" will be executed interactively. If you can confirm this, you are successful!

Summary

This time, with the motivation to write an article on Qiita, I introduced how to easily set up a clean and reproducible python execution environment, and even introduced how to execute it interactively with VS Code. Even if you are not writing a blog, it is convenient to be able to prepare multiple independent python environments with a single command. After all, I realized again that Docker is amazing (vocabulary). In the future, I would like to use this environment to write some articles that are valuable in terms of data science. Thank you for watching until the end! !!

Recommended Posts

Write about building a Python environment for writing Qiita Qiita
[Mac] Building a virtual environment for Python
Building a Python development environment for AI development
How about Anaconda for building a machine learning environment in Python?
Think about building a Python 3 environment in a Mac environment
Building a Python virtual environment
Building a Python virtual environment
Building a Python environment for pyenv, pyenv-virtualenv, Anaconda (Miniconda)
Building a Docker working environment for R and Python
Procedure for building a CDK environment on Windows (Python)
Building a Python environment for programming beginners (Mac OS)
Memo for building a machine learning environment using Python
Building a Python environment on Mac
Building a Python environment on Ubuntu
Building a virtual environment with Python 3
Building a python environment for artificial intelligence (Chainer / TensorFlow / CSLAIER)
Building a development environment for Android apps-creating Android apps in Python
Building a Hy environment for Lisper who hasn't touched Python
[Python] Building a virtual python environment for the pyramid tutorial (summary)
[Pyenv] Building a python environment with ubuntu 16.04
Building a Python3 environment with Amazon Linux2
Building a Docker working environment for R and Python 2: Japanese support
Building a Windows 7 environment for getting started with machine learning with Python
Let's create a virtual environment for Python
From building a Python environment for inexperienced people to Hello world
Building a Python 3.6 environment with Windows + PowerShell
Building a conda environment for ROS users
Building a virtual environment for Mayavi dedicated to Python 3.6, Anaconda, Spyder users
Selenium + WebDriver (Chrome) + Python | Building environment for scraping
Created a Python wrapper for the Qiita API
Building a python environment with virtualenv and direnv
Building an environment for executing Python scripts (for mac)
Building an Anaconda environment for Python with pyenv
Building a Python environment with WLS2 + Anaconda + PyCharm
A memo about writing merge sort in Python
[Python] Web development preparation (building a virtual environment)
Commands for creating a python3 environment with virtualenv
Procedure for creating a Python quarantine environment (venv environment)
A memo for creating a python environment by a beginner
Building a Python environment on a Sakura VPS server
About Python for loops
About Python, for ~ (range)
About Python development environment
Create a Python environment
Python environment for projects
Recommendation of building a portable Python environment with conda
Building a training environment for penetration testing using Naumachia
Build a python environment for each directory with pyenv-virtualenv
Building an environment for natural language processing with Python
Building and enabling a python virtual environment, etc. (venv)
A memo about building a Django (Python) application with Docker
conda memorandum: Building a Python environment with supercomputer ITO
python memo (for myself): About the development environment virtualenv
About Fabric's support for Python 3
Python environment construction For Mac
Python3 environment construction (for beginners)
A memorandum about correlation [Python]
A memorandum about Python mock
[For organizing] Python development environment
Build a Python environment offline
About "for _ in range ():" in python