Build PyPy and Python execution environment with Docker

This is a sequel (sister edition?) Of the previous article "Building a PyPy execution environment with Docker". Last time we only had a PyPy environment, but this time we will create an environment for both PyPy and plain Python. Please note that it contains a lot of the same content.

Introduction

What is PyPy?

One of the Python implementations. You can run Python modules normally. It runs much faster than plain Python.

Why PyPy and Python?

The downside of PyPy is that it limits the libraries available. I don't know the details, but it seems that third-party libraries are often unavailable. Therefore, it is convenient to be able to use plain Python for the part that uses the third-party library in the series of processing, and execute it with PyPy for the other part that requires execution speed. This time, we will build an environment that can use both PyPy and Python with Docker.

Why Docker?

I wanted to create a virtual environment because I didn't want to pollute the environment of the local machine, but I couldn't find much information on how to create a virtual environment with PyPy. Since I was studying Docker recently and I don't really trust the Python environment of Windows in the first place, I decided to create an environment with Docker.

environment

--Windows 10 Home (version 20H2 build 19042.685) --Environment where Docker can be used with WSL2

Recently, Docker has started to work on WSL2 even on Windows 10 Home, making it very easy to use. For details on how to install WSL2, refer to Microsoft Official Page, for example. After that, you can specify to use WSL2 in the Docker Desktop settings.

You should be able to do it on your Mac in much the same way.

procedure

policy

The basic policy is as follows.

--Build a PyPy and Python environment on a container based on PyPy's official Docker image --PyPy environment is managed by Pip and requirements.txt --Pipenv manages Python environment

The reason for using PyPy's Docker image instead of Python is simply because it's the easiest way to set up a PyPy environment. Moreover, the PyPy image also includes a Python execution environment. To avoid cluttering the environment, we will manage the libraries used in Python in the virtual environment of Pipenv. Keep the pypy environment in requirements.txt to make it easier to rebuild and share the environment.

Directory structure

Configure the directory as follows:

│  docker-compose.yml
│  Dockerfile
└─src
        pypy_main.py        
        python_main.py
        Pipfile
        Pipfile.lock
        requirements.txt

File creation

In requirements.txt, describe the library you want to use in the pypy environment. Here, numpy is specified as an example.

requirements.txt


numpy

Pipfile and Pipfile.lock specify the library you want to use in your Python environment. Here, as an example, we will prepare one that installs only pandas. Also note that we have specified 3.9 as the Python version (will come out later).

Pipfile.


[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pandas = "*"

[dev-packages]

[requires]
python_version = "3.9"

pypy_main.py is the module you want to run in the pypy environment. Here, as an example, prepare something to execute numpy.

pypy_main.py


import numpy as np

print(np.array([1, 2, 3]))

pyhon_main.py is the module you want to run in the Python environment. Here, as an example, we will prepare something to run pandas.

python_main.py


import pandas as pd

print(pd.DataFrame([[1, 2, 3], [4, 5, 6]]))

Create Dockerfile as follows.

# 1.Get PyPy image
FROM pypy:3.7

# 2.pyenv and python3.9 installation
RUN git clone https://github.com/pyenv/pyenv.git ~/.pyenv && \
    echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc && \
    echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc && \
    echo 'eval "$(pyenv init -)"' >> ~/.bashrc && \
    . ~/.bashrc  && \
    pyenv install 3.9.1

# 3.Work directory/Specified in src
WORKDIR /src

# 4.python virtual environment construction
COPY src/Pipfile* ./
RUN pip install pipenv && \
    pipenv install

# 5.pypy environment construction
COPY src/requirements.txt .
RUN pip install -r requirements.txt
  1. Get the official image of PyPy from Docker Hub.
  2. Install Python 3.9 using pyenv. This is because version 3.9 is specified in the Pipfile. Reference → https://codeaid.jp/pyenv-linux/
  3. Specify the work directory in/src for further processing.
  4. Build a Python virtual environment using Pipenv. I am running the pipenv install command after copying the Pipfile and Pipfile.lock.
  5. Copy requirements.txt and install the required libraries from it to build your PyPy environment.

Create the docker-compose.yml file as follows.

docker-compose.yml


version: '3'

services:
  pypy-python:
    build: .
    volumes:
      - ./src/:/src
    tty: true

The service name is pypy-python, but anything is fine. You will enter the container with this name later, so I think you should name the project so that it is easy for you to understand. Dockerfile in the same directory is specified in build. The src directory of the host PC is mounted on the src directory of the container with volumes. The edits on the host PC will now be reflected in the container. By setting tty to true, you can enter the container by keeping it running after launching the container.

The preparation is complete.

Run

Open a command prompt in the directory containing docker-compose.yml and start the container with the following command.

>docker-compose up -d

The first time will take some time. By adding the option -d, it will be executed in the background, and subsequent command operations will be possible.

Once the container is up, enter it with the following command.

>docker-compose exec pypy-python bash

Here, the argument pypy-python is the service name given in the docker-compose.yml file. The argument bash is needed to perform command line operations after entering the container. Or rather, an error will occur without this argument.

Now that we're in the src directory inside the container, we have pypy_main.py and python_main.py directly underneath.

Use the pypy command to run PyPy. Try running pypy_main.py.

# pypy pypy_main.py
[1 2 3]

You can do it like this. You can also see that the numpy described in requirements.txt can be installed in the pypy environment.

Next, try running python_main.py in the virtual environment of Pipenv.

# pipenv run python python_main.py
   0  1  2
0  1  2  3
1  4  5  6

As you can see, I was able to run python in the Pipenv environment. You can see that pandas can be used.

Edit source

Since the src directory is mounted, if you edit the file on the host PC, it will be reflected in the container. Development can proceed in the flow of editing with the editor of the host PC and executing in the container.

Library management

The libraries in the pypy environment can be installed with pip. For example, execute the following command in the container to install tqdm.

# pip install tqdm

After installing the library, it is a good idea to record it in requirements.txt.

# pip freeze > requirements.txt

Since the src directory is mounted, if you rewrite it in the container, it will be reflected on the host machine. If you save this file, you can reproduce the same environment when you recreate the container.

Of course, the library of Python environment is managed by Pipenv. If you want to install a new library, do pipenv install inside the container. The rewritten Pipfile and Pipfile.lock are also reflected on the host machine, so if you save this, you can reproduce the same environment.

End of container

When you are done using it, use the exit command to exit the container.

# exit

You can leave the container up, but I think it's a good idea to drop it when you're done using it.

>docker-compose down

To use it, do docker-compose up -d again. From the second time onward, the image remains on the host PC, so it will start up quickly. However, if you docker-compose down after installing a new library in the container, the host PC image will not reflect the change. In order for the library installation to take effect, you need to rebuild by adding the --build option as shown below. This will load a new Pipfile and reqruirements.txt and recreate the container from the image.

>docker-compose up -d --build

in conclusion

I am creating a Unexplored Village Search Tool, and I am using PyPy and Python together as introduced this time in the data preprocessing of this. I had to do a lot of calculations after reading the GIS data, but the library (geopandas) that reads the GIS data cannot be used with PyPy, and the calculation process never ends with Python. Therefore, I took the method of reading GIS data with Python, outputting it to a text file, and performing calculation processing with PyPy. I think PyPy is fast and convenient, but I have the impression that there is little information. Isn't there a lot of people using it?

Also, Docker is convenient. If you use it well, you will not have to worry about the environment and it will be very comfortable. (In fact, why doesn't Python work so well in a Windows environment these days ...)

Recommended Posts

Build PyPy and Python execution environment with Docker
Build PyPy execution environment with Docker
Build Mysql + Python environment with docker
Build Jupyter Lab (Python) environment with Docker
Create a C ++ and Python execution environment with WSL2 + Docker + VSCode
How to build Python and Jupyter execution environment with VS Code
Prepare the execution environment of Python3 with Docker
[Python] Build a Django development environment with Docker
Build a python execution environment with VS Code
Build python3 environment with ubuntu 16.04
Prepare python3 environment with Docker
Build python environment with direnv
Build a python virtual environment with virtualenv and virtualenvwrapper
Build python virtual environment with virtualenv
Build a Python + bottle + MySQL environment with Docker on RaspberryPi3! [Trial and error]
Build GPU environment with GCP and kaggle official image (docker)
Build Python3 and OpenCV environment on Ubuntu 18.04
Create Python + uWSGI + Nginx environment with Docker
[Linux] Build a jenkins environment with Docker
Build a python virtual environment with pyenv
Clean python environment with pythonz and virtualenv
Build a modern Python environment with Neovim
Build AI / machine learning environment with Python
Build NGINX + NGINX Unit + MySQL environment with Docker
[Linux] Build a Docker environment with Amazon Linux 2
Build a 64-bit Python 2.7 environment with TDM-GCC and MinGW-w64 on Windows 7
Build a Python environment on your Mac with Anaconda and PyCharm
Build and try an OpenCV & Python environment in minutes using Docker
Build a Python execution environment using GPU with GCP Compute engine
Create a simple Python development environment with VS Code and Docker
[DynamoDB] [Docker] Build a development environment for DynamoDB and Django with docker-compose
Recommended environment and usage when developing with Python
Build python environment with pyenv on EC2 (ubuntu)
Simply build a Python 3 execution environment on Windows
Build a Python + bottle + MySQL environment with Docker on RaspberryPi3! [Easy construction]
Build Python development environment with Visual Studio Code
Go (Echo) Go Modules × Build development environment with Docker
Build a python environment with ansible on centos6
Create a python3 build environment with Sublime Text3
I set the environment variable with Docker and displayed it in Python
Create Nginx + uWSGI + Python (Django) environment with docker
Build a CentOS Linux 8 environment with Docker and start Apache HTTP Server
Build a virtual environment with pyenv and venv
Use python with docker
Python environment with docker-compose
Try using tensorflow ① Build python environment and introduce tensorflow
Build a Python environment with OSX El capitan
Quickly build a Python Django environment with IntelliJ
Virtual environment with Python 3.6
Build a Python machine learning environment with a container
Build a development environment using Jupyter and Flask with Python in Docker (supports both VS Code / code-server)
Build a local development environment with WSL + Docker Desktop for Windows + docker-lambda + Python
Install Python3 on Mac and build environment [Definitive Edition]
Japanese can be used with Python in Docker environment
Building a Docker working environment for R and Python
[Python] OpenCV environment construction with Docker (cv2.imshow () also works)
Build a python environment for each directory with pyenv-virtualenv
Use Python installed with pyenv for PL / Python execution environment
How to build a Django (python) environment on docker
Build a machine learning application development environment with Python
Preparing the execution environment of PyTorch with Docker November 2019