Unify the environment of the Python development team starting with Poetry

Unify the environment of the Python development team starting with Poetry

Overview

It was when I was asked to look into the environment of a certain development site. The ** environment construction procedure is given verbally **, and Anaconda is installed on Windows and additional packages are installed with pip. Of course, ** version is not specified **, so the latest version at that time will be included, so the minor version will differ depending on the entry timing. In a large team that straddles companies, it is possible that company A works but company B does not. ** Since the introduction of Lint and formatter is optional **, the commit log of git ** the diff of the added and modified part and the formatted part coexist **, and the source review takes time.

Would you like to unify the Python operating environment for all members of the development team? Would you like to create a clean source that is easy to maintain according to the code rules?

Let's start the work style reform of engineers with ** Poetry **!

goal

  1. Understand how to code the environment using Poetry and how to distribute it to team members.
  2. Get the project template and get a smooth start to development.

We will aim for the above. By the way, here are the main problems of the development site mentioned in the overview.

problem What's wrong Will Poetry solve it?
Oral environment construction procedure(Or txt) Causes human error such as omission Reproducibility because the environment installer is completed ◎
Pip install on Anaconda environment In the worst case, you need to reinstall the Anaconda environment. Don't worry because you don't use Anaconda
No version specified It doesn't work depending on the person ... Version can be unified with one command
Lint and formatter are optional Hard-to-read code is hard to maintain It can be mandatory to install the tool.
Autorun needs to be combined with vscode and git.

procedure

1. Installation of poetry

We will install poetry under the python environment. It is a one-shot installation with the following command. There is no need to learn operations like Docker.

$pip install poetry

2. Create project template

After installing poetry, we will create a template for the project.

$ poetry new xxx

The part corresponding to xxx is the project name. This is the part of * pip install xxx * that corresponds to xxx. You will have a source / test directory and a Readme, pyproject.toml file. After that, push to the git repository and the initial setting is completed.

When executed, it looks like this
ezgif.com-gif-maker.gif

3. Dependent library settings

Edit the ** pyproject.toml ** file to configure the dependent libraries required by your project. pyproject.tomlはパッケージビルドに必要なデータを定義するファイルのフォーマットで、setup.py/cfgを置き換えてsetuptoolsではないビルドツールでもビルドが可能となることを目的として作られたものです。

#Add dependent library
$ poetry add xxx
#Added development library
$ poetry add -D xxx

Run!

You can run it just like pip install. If you want to specify the version, use the == operator.

$ root@16460604cb93:~/poetry_template# poetry add tensorflow==2.3.0
Creating virtualenv poetry-template-oDQfK0qA-py3.8 in /root/.cache/pypoetry/virtualenvs

Updating dependencies
Resolving dependencies... (48.4s)

Writing lock file

Package operations: 41 installs, 0 updates, 0 removals
$ root@16460604cb93:~/poetry_template# poetry add -D mypy black isort
Using version ^0.790 for mypy
Using version ^20.8b1 for black
Using version ^5.6.4 for isort

Execution result

As you execute the add command, it will be added to the pyproject.toml file. You can also edit pyproject.toml directly.

[tool.poetry]
name = "poetry_template"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"
tensorflow = "2.3.0"

[tool.poetry.dev-dependencies]
pytest = "^5.2"
mypy = "^0.790"
black = "^20.8b1"
isort = "^5.6.4"
flake8 = "^3.8.4"
radon = "^4.3.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

4. Create an execution environment

We will create a virtual environment (venv) in which the environment according to the conditions described in the pyproject.toml file is installed.

Change the location where venv is created

By default, venv is created in the home directory, but it would be nice if it could be created directly under the project, such as when using it with vscode, so change the setting.

$ poetry config virtualenvs.in-project true
$ poetry config --list

Install environment

Installation is also a command. You will have a .venv directory and a poetry.lock file.

$ poetry install
When executed, it looks like this
ezgif.com-gif-maker (1).gif

The poetry.lock file is a file that guarantees full dependencies. Even if you didn't specify some explicit versions in the pyproject.toml file, the lock file records version information for dependencies that have been successfully installed in the past so that anyone can reproduce them at any time. In other words, if you ** distribute this file to team members ** and have them ** poetry install **, you will always have a unique environment.

5. Run Python in the environment you created

We will access the virtual environment created last. You can access the commands in the virtual environment with the poetry run command.

#Attach to python
$ poetry run python -V
Python 3.8.5
#Test run
$ poetry run pytest
======================================================= test session starts =======================================================
platform linux -- Python 3.8.5, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /root/poetry_template
collected 1 item

tests/test_poetry_template.py .                                                                                             [100%]

======================================================== 1 passed in 0.01s ========================================================

Actually usable project template

It's been a long time, but that's all for poetry. I hope you understand that you can code the environment with just a few commands, manage it with Git, and all project members can develop in the same environment.

We have created a project template that you can actually use, so please use it. https://github.com/TomokiHirose/poetry_template

Template overview

I am creating a project template with the following functions. Rewrite the template and use the poetry update command to rewrite the environment to suit the user.

command Contents
poetry run pytest ./Run the tests using the file under tests
poetry run format black(Powerful formatter)And isort(Arrangement of import order)Runs
poetry run lint flake8(linter)And mypy(Static type checking)Runs
poetry run metrics Performs static analysis of cyclomatic complexity index, maintenance availability index, LOC, etc.
poetry run apidoc Generate API documentation with sphinx
poetry run testdoc Generate Test document with sphinx[^1]
poetry build ./wheel file under dist(python installer)To generate

in conclusion

I forgot to install it! Even in that case, instead of "Please install xx", just push the fix to git and all the members hit the update command, and everyone will be in the same environment immediately! Don't you think it's very modern?

Furthermore, in cooperation with vscode, if black and flake8 under .venv are automatically executed at the time of saving, or if black is run before push by linking with the commit hook of git, the format is forced. can do. I don't think it's great to follow PEP8 [^ 2], but if you force ** anyone to write the same code **, even if advanced and newcomers are in the same project It is recommended because it will be easier to maintain because the code like this will be completed.


[^ 1]: Document that records the items to be confirmed in the test and the tester in charge. [^ 2]: PEP8 is a Python coding convention. It is not always necessary to follow it, but there is a difference in ease of maintenance depending on whether there is one standard or not.

Recommended Posts

Unify the environment of the Python development team starting with Poetry
Prepare the execution environment of Python3 with Docker
Get a quick Python development environment with Poetry
Manage Python runtime packages and development environment packages with Poetry
Get a clean Python development environment with pyenv + pipx + Poetry
How to get into the python development environment with Vagrant
Checking the NAOqi Python development environment
Prepare Python development environment with Atom
Prepare the development environment with anyenv
[Development environment] Python with Xcode [With screen transition]
Check the existence of the file with python
About the virtual environment of python version 3.7
The strongest Python development environment PyCharm's recommendation
Windows + gVim + Poetry python development environment construction
The strongest Python integrated development environment PyCharm
Note: Prepare the environment of CmdStanPy with docker
Build the fastest Django development environment with docker-compose
2016 The University of Tokyo Mathematics Solved with Python
Build Python development environment with Visual Studio Code
[Note] Export the html of the site with python.
Calculate the total number of combinations with python
Instantiation of the BOX development environment created earlier
[Python] Build a Django development environment with Docker
Use multiple versions of python environment with pyenv
Check the date of the flag duty with Python
Image processing? The story of starting Python for
Poetry-virtualenv environment construction with python of centos-sclo-rh ~ Notes
python development environment -use of pyenv and virtualenv-
Convert the character code of the file with Python3
Continuation of multi-platform development with Electron and Python
[Python] Determine the type of iris with SVM
I installed Pygame with Python 3.5.1 in the environment of pyenv on OS X
Unification of Python environment
Python development environment construction
the zen of Python
Python starting with Windows 7
About Python development environment
GRPC starting with Python
Python environment with docker-compose
python2.7 development environment construction
Development environment in Python
Virtual environment with Python 3.6
Virtualize (isolate) IBM i python development environment with chroot
Extract the table of image files with OneDrive & Python
Learn Nim with Python (from the beginning of the year).
Recommendation of building a portable Python environment with conda
The story of sharing the pyenv environment with multiple users
Destroy the intermediate expression of the sweep method with Python
Create a python development environment with vagrant + ansible + fabric
Visualize the range of interpolation and extrapolation with python
Calculate the regression coefficient of simple regression analysis with python
Build a machine learning application development environment with Python
Python development environment with Windows + Anaconda3 + Visual Studio Code
Python development environment construction 2020 [From Python installation to poetry introduction]
Preparing the execution environment of PyTorch with Docker November 2019
QGIS3 Python plugin development environment construction with VSCode (macOS)
Summary of the basic flow of machine learning with Python
Python development environment with Windows + Python + PipEnv + Visual Studio Code
Install Ubuntu 20.04 with GUI and prepare the development environment
Get the operation status of JR West with Python
Build a development environment with Poetry Django Docker Pycharm