It's been a while since I started programming in Python, and I've understood the directory structure, so I'll summarize it. At first, I didn't understand it at all, and I was googled quite a bit, so I wrote it in steps so that even beginners could understand it.
The following is a memo by an engineer who has been developing in Python for about half a year. ** We would be grateful if you could give us a lot of suggestions and suggestions for better methods, and questions from beginners. ** **
When I first touched Python, I wrote a lot of methods in hoge.py for the time being, and eventually there were multiple files and more directories. However, if I did this, I failed to resolve the dependencies in my program, and the structure became more and more incomprehensible, and I felt it was difficult to correct it later.
Therefore, unless it is a very urgent programming, I think it is important to first ** "prepare directories and files so that it can be easily made into a library" **. I will create the template in this article.
First, create a python_package directory as your project folder and create a new setup.py. Naturally, the area directly under the project is as shown below.
.
└── setup.py
** setup.py is absolutely necessary when you provide your own program as a library. ** **
For the time being, let's fill in what can be put in at present. You decide the name of the project you are trying to create and the location of the source.
setup.py
from setuptools import setup
setup(
name="koboripackage",
version='1.0',
description='For testing Python directory structure',
author='Kobori Akira',
author_email='[email protected]',
url='https://github.com/koboriakira/python_package',
)
If the product to be created is clear, you can create a README.md at this stage.
After writing setup.py, the next step is to prepare the source directory.
Create a directory with the same name as you wrote in name of setup, and put \ _ \ _ init \ _ \ _.py in it for the time being. An empty file is OK.
Also create a Python file with the same file name as the directory name. Let's say this is a module used as a library.
.
├── koboripackage
│ ├── __init__.py
│ └── koboripackage.py
└── setup.py
koboripackage.py
def hello(name):
print('Hello, %s!' % name)
Once created, run python setup.py sdist in the directory containing setup.py.
$ python setup.py sdist
running sdist
running egg_info
.
.
.
creating dist
Creating tar archive
removing 'koboripackage-1.0' (and everything under it)
Then, various files increased.
.
├── dist
│ └── koboripackage-1.0.tar.gz
├── koboripackage
│ ├── __init__.py
│ └── koboripackage.py
├── koboripackage.egg-info
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ ├── dependency_links.txt
│ └── top_level.txt
└── setup.py
Of these, the tar in dist is a library that everyone can use with pip install etc.
We haven't published the library yet, so for now let's specify this tar directly and put it locally.
pip install dist/koboripackage-1.0.tar.gz
Let's see if it can actually be used as a library. Run Python in the CLI.
$ python
>>> from koboripackage import koboripackage
>>> koboripackage.hello('World')
Hello, World!
>>>
In this way, I was able to import and use the modules and functions I created.
You can uninstall the library with pip uninstall koboripackage.
Libraries made in Python can also be run from the terminal on the command line.
Let's add a setting for that. First, create cli.py directly under the source directory.
cli.py
def execute():
print('The command has been executed!')
Then edit setup.py.
version to 1.1find_packages to import and addpackages = find_packages (). The format is written with command name = module: method`setup.py
from setuptools import setup, find_packages
setup(
name="koboripackage",
version='1.1',
description='For testing Python directory structure',
author='Kobori Akira',
author_email='[email protected]',
url='https://github.com/koboriakira/python_package',
packages=find_packages(),
entry_points="""
[console_scripts]
koboripackage = koboripackage.cli:execute
""",
)
Run python setup.py sdist again when you're done.
Then, a new v1.1 tar is created in dist, and the configuration is as follows.
.
├── dist
│ ├── koboripackage-1.0.tar.gz
│ └── koboripackage-1.1.tar.gz
├── find_package.txt
├── koboripackage
│ ├── __init__.py
│ ├── cli.py
│ └── koboripackage.py
├── koboripackage.egg-info
└── setup.py
As before, try installing the library with pip install dist / koboripackage-1.1.tar.gz. Then
$ koboripackage
The command has been executed!
You can see that we were able to execute the execute method of cli.py with just the command name as described above.
It may be wrong, but I will write down my understanding.
find_packages () is a method to find the source you need when creating a tar with python setup.py sdist.
And you will need __init__.py for the search.
Only in this chapter we will add the sub directory and sub_module.py to the source directory.
.
├── koboripackage
│ ├── __init__.py
│ ├── cli.py
│ ├── koboripackage.py
│ └── sub
│ ├── __init__.py
│ └── sub_module.py
├── setup.py
If there is no \ _ \ _ init \ _ \ _. Py in the sub directory at this time, find_packages () will not find the files in and in the sub directory.
You can check which file you are finding = packaged in SOURCES.txt in egg-info.
$ cat koboripackage.egg-info/SOURCES.txt
MANIFEST.in
requirements.txt
setup.py
koboripackage/__init__.py
koboripackage/cli.py
koboripackage/koboripackage.py
koboripackage.egg-info/PKG-INFO
koboripackage.egg-info/SOURCES.txt
koboripackage.egg-info/dependency_links.txt
koboripackage.egg-info/entry_points.txt
koboripackage.egg-info/requires.txt
koboripackage.egg-info/top_level.txt
koboripackage/sub/__init__.py
koboripackage/sub/sub_module.py
tests/__init__.py
Try importing an existing library into cli.py.
cli.py
import requests
def execute():
print('The command has been executed!')
response = requests.get('https://www.google.com/')
print(response.status_code)
To use requests (locally), you need to import the library with pip install requests.
Similarly, when providing it as a library, it is necessary to specify "what must be imported".
Prepare requirements.txt for this. Place it directly under the project.
requirements.txt
requests==2.23.0
By the way, you can check the required library and version with the pip freeze command.
However, all the libraries currently installed are output, so if you copy and paste as it is, you may include unnecessary libraries.
Next, make two settings so that requirements.txt is used correctly when creating a tar with python setup.py sdist.
.
├── MANIFEST.in
├── dist
├── koboripackage
│ ├── __init__.py
│ ├── cli.py
│ └── koboripackage.py
├── koboripackage.egg-info
├── requirements.txt
└── setup.py
MANIFEST.in
include requirements.txt
setup.py
from setuptools import setup, find_packages
setup(
name="koboripackage",
version='1.2',
description='For testing Python directory structure',
author='Kobori Akira',
author_email='[email protected]',
url='https://github.com/koboriakira/python_package',
packages=find_packages(),
entry_points="""
[console_scripts]
koboripackage = koboripackage.cli:execute
""",
install_requires=open('requirements.txt').read().splitlines(),
)
** MANIFEST.in can "add something that is not normally included when packaged (and vice versa)" **. Since I wrote ʻinclude requirements.txt`, I will add requirements.txt when packaging.
Then, in ʻinstall_requires` added to setup, specify the required libraries. You can specify the library name directly as an array, but it seems that it is common to read the contents of requirements.txt as described above.
install_requires=['requests']
When you are done so far, create a v1.2 tar with a file with python setup.py sdist.
When I run the command after installing the new version of the library with pip install dist / koboripackage-1.2.tar.gz,
$ koboripackage
The command has been executed!
200
You can see that the library is available as above.
If you have prepared so far, you can register for PyPi (Python Package Index).
Please refer to https://blog.amedama.jp/entry/2017/12/31/175036 for the specific procedure.
If you can upload it successfully, you can install it with pip install koboripackage etc.
Be prepared for testing.
Create a tests folder directly under the project. Create a Python file for testing with the same structure as the directory containing the source (add test to the prefix). Don't forget \ _ \ _ init \ _ \ _. Py.
.
├── MANIFEST.in
├── dist
├── koboripackage
│ ├── __init__.py
│ ├── cli.py
│ └── koboripackage.py
├── koboripackage.egg-info
├── requirements.txt
├── setup.py
└── tests
├── __init__.py
└── test_koboripackage.py
Write the source as follows for koboripackage.py and test_koboripackage.py, and try to check the behavior of multiply ().
koboripackage.py
def hello(name):
print('Hello, %s!' % name)
def multiply(a, b):
return a * b
test_koboripackage.py
from koboripackage import koboripackage
def test_multiply():
assert koboripackage.multiply(2, 3) == 6
Install pytest with pip install pytest, then run pytest.
$ pytest
================================================================================================== test session starts ==================================================================================================
platform darwin -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: ...............
collected 1 item
tests/test_koboripackage.py . [100%]
=================================================================================================== 1 passed in 0.02s ===================================================================================================
You have now created a general Python project.
Finally, I will prepare a Docker image because it is a big deal. After adding the necessary files, create an image with the library installed in advance.
FROM python:3.7.5-slim
WORKDIR /work
ADD koboripackage koboripackage
ADD tests tests
ADD requirements.txt requirements.txt
ADD setup.py setup.py
ADD MANIFEST.in MANIFEST.in
RUN pip install --upgrade pip \
&& pip install -r requirements.txt \
&& pip install pytest
CMD bash
Create an image with docker build -t python_package . and create a container with docker run -it --rm python_package.
You can now run pytest inside a Docker container.
If you want to develop it as it is, you can share the source as follows.
docker run -it --rm -v $(pwd)/koboripackage:/work/koboripackage -v $(pwd)/tests:/work/tests python_package
Finally push this to github. Some of the files created during this process do not need to be uploaded to github, so prepare .gitignore.
.gitignore
**/__pycache__
dist/
koboripackage.egg-info/
The package created in this way is as follows. I also uploaded it to Ichiou PyPi. https://github.com/koboriakira/python_package
Originally, I would write the license in setup.py, but I didn't have the confidence to enter, so I avoided it this time. At present, we are only aware of "I wonder if it should be MIT for the time being".
By preparing a state that can be made into a library in this way from the beginning, I hope that it will be easier to consciously create components of appropriate size.
Recommended Posts