What is the XX file at the root of a popular Python project?

Preface

I want to understand * setup.py *, * requirements.txt *, and other files that I often see in Python projects. What kind of files are there in the first place? I was researching various things. For the time being, I was looking at the contents of a popular Python project on GitHub to imitate my predecessor, and there are configuration files for various tools, and there are such tools. Shall we? I became interested in something different from the original purpose.

That's why ** about 700 projects with the most stars in the Python project on GitHub ** are counted in the root file and the most used files are aggregated, what tool is used? I looked it up.

rule

• Sort by "Most Starts" from Python Topic on GitHub and search all repositories available on this page. The total number is ** 689 **.
• Describe files that are likely to be used frequently, and describe the results of a simple examination if you do not understand
• Exclude the following files
• git related files
• • .gitignore *, * .gitattributes *, etc.
• Document system
• • README.md *, * CONTRIBUTING.md *, etc.
• The ones that are not used very often
• Other project-specific items, etc.
• Total counts for items with notational fluctuations
• • codecov.yml * and * .codecov.yml * etc.
• All the searched results are in Gist, so if you are interested, please.
• Welcome to point out incorrect explanations, captures, missing files, etc.

A system often seen in Python projects

setup.py

◆ Number of projects used: 503 ◆ Tools used: setuptools

Describes all behavior when building, distributing, and installing a Python project as a library. Since the directory where * setup.py * is located is recognized as a Python project, you can pip install by specifying the local directory where * setup.py * is located, the local zip file, the Git repository, etc. Will be.

pip install path/prj/dir
pip install path/dir/file.zip
pip install git+https://github.com/{user-name}/{repo-name}

pip installs the package according to the description in setup.py.

To package the project as a library, specify python setup.py sdist or python setup.py bdist_wheel and execute.

#When creating a source code distribution
python setup.py sdist

#Wheel required to create a pre-built distribution
pip install wheel
python setup.py bdist_wheel

Reference: Building and Distributing Packages with Setuptools

MANIFEST.in

◆ Number of projects used: 402 ◆ Tools used: setuptools

If you want to include files other than source code in the distribution, write them in this file. If you write ʻinclude_package_data = True in the argument (or * setup.cfg *?) Of Setup () `, this file will be processed.

You can specify the file with more detailed conditions by specifying the package_data parameter.

Reference: https://setuptools.readthedocs.io/en/latest/setuptools.html#including-data-files

setup.cfg

◆ Number of projects used: 344 ◆ Tools used: setuptools

When packaging the project, I had to write various things in the arguments of the setup () function of * setup.py *. However There were problems such as being difficult to write and difficult to interpret by tools other than * setuptools *. Therefore, * setup.cfg * was created so that a series of settings can be put together in a separate file.

This eliminates the need to write arguments to setup (). (However, if you use an old build tool, you need to describe the argument)

reference

• Configuring setup() using setup.cfg files
• PEP 517 -- A build-system independent format for source trees
• PEP 518 -- Specifying Minimum Build System Requirements for Python Projects

requirements.txt

◆ Number of projects used: 261 ◆ Tools used: pip

• setup.py * and * setup.cfg * cannot specify the version of dependent packages. Therefore, the file that describes the version of the dependent package is * requirements.txt *. However, the file name is arbitrary, so you can use a different name.

Type pip freeze on the command line to output dependency information on the command line. If you set pip freeze> requirements.txt, it will be output to the file as it is. If you are in the virtualenv (venv) environment, the packages in virtualenv will be output. The packages output by freeze do not include packages for package management such as * pip *, * setuptools *, and * wheel *.

requirements.txt example

appdirs==1.4.3
astroid==2.3.3
botocore==1.15.6

Abbreviation

When installing the dependency, it looks like the following

pip install -r requirements.txt

How to write: https://pip.pypa.io/en/latest/reference/pip_install/#example-requirements-file

The tool used is mainly pip, but Pipenv can also install packages from * requirements.txt *, and conversely generate * requirements.txt * from * Pipfile *. Doesn't Poetry seem to be possible?

pyproject.toml

◆ Number of projects used: 102 ◆ Tools used: Poetry, flit, [setuptools](https://github. com / pypa / setuptools), Pyflow, black etc.

• pyproject.toml * is a format for describing various information and dependencies for packaging instead of * setup.py * and * setup.cfg *, which depend on * setuptools *, [PEP 518]( It is formulated at https://www.python.org/dev/peps/pep-0518/). You can also use * pyproject.toml * to replace files such as * requirements.txt *, * MANIFEST.in *, and * Pipfile *.

If the package you want to install with pip install is distributed in sdist format and you need to build it yourself If the pip version is v19.0 or later and the target package has a * pyproject.toml * file, the package will be built according to PEP 517. Building packages using * pyproject.toml * is like Poetry, Pyflow Tools are supported, but it seems that * setuptools * is also supported in recent versions.

Build in the \ [tool ] table as described in the tool table section of PEP 518 Can describe tool settings for Python projects other than tools. You need to use the subtables in \ [tool ], [tool.flit.metadata], black for flit In the case of (: //github.com/psf/black), you can describe the settings of each tool by writing as [tool.black].

• Reference
• setuptools: setup.cfg-only projects
• PEP 453 -- Explicit bootstrapping of pip in Python installations
• PEP 517 -- A build-system independent format for source trees
• PEP 518 -- Specifying Minimum Build System Requirements for Python Projects

requirements-dev.txt

◆ Number of projects used: 51+ ◆ Tools used: pip

Requirements.txt for development / test environment. As mentioned above, the file name of * requirements.txt * is arbitrary, so Dependent package information used only in the test environment is saved under a different name from * requirements.txt *. However, there are quite a lot of name fluctuations, and there are other:

• dev-requirements.txt
• test-requirements.txt
• requirements_dev.txt
• requirements-test.txt
• requirements_test.txt
• docs-requirements.txt
• test_requirements.txt
• requirements-docs.txt
• requirements_docs.txt
• optional-requirements.txt
• requirements-build.txt

And so on.

Pipfile

◆ Number of projects used: 26 ◆ Tools used: Pipenv

A file that describes the dependent package information used by Pipenv.

• Pipfile.lock * is created based on this file. In addition to the dependent package information, you can also write scripts etc. and execute it like pipenv run script-name.

Pipfile.lock

◆ Number of projects used: 22 ◆ Tools used: Pipenv

Pipenv A lock file that describes the information of the generated dependent packages. Something like * package-lock.json * in npm. Since the package described in the lock file is installed as it is with pipenv sync (pipenv sync -d if dev-package is also installed), the same environment can be reproduced among multiple users in the development environment / operation environment. ..

There was also one * pipfile.lock *. (Can it be used?)

It doesn't match the number of * Pipfile *, but everyone, let's manage the lock file properly with Git. I thought ... but according to the Pipenv documentation, "Targeting multiple versions of Python" If so, don't version control the lock file. "

• Generally, keep both Pipfile and Pipfile.lock in version control.
• Do not keep Pipfile.lock in version control if multiple versions of Python are being targeted.

poetry.lock

◆ Number of projects used: 15 ◆ Tools used: Poetry

A lock file that describes the information of dependent packages generated by Poetry. It is generated based on the contents described in * pyproject.toml *.

Other Python libraries

tox.ini

◆ Number of projects used: 225 ◆ Tools used: tox

A virtualenv management and command line tool for testing with multiple Python versions and interpreters. Write the settings to * tox.ini *, but it seems that it also supports * pyproject.toml *

Documentation

.coveragerc

◆ Number of projects used: 189 ◆ Tools used: coverage

Configuration file for test coverage measurement tools.

Documentation

codecov.yml

◆ Number of projects used: 102 ◆ Tools used: codecov

This is also a configuration file for another coverage measurement tool. It's less than * .coveragerc *, but it's also used in quite a few projects. Also with * .codecov.yml *.

Documentation

.pylintrc

◆ Number of projects used: 80 ◆ Tools used: pylint

The configuration file for the linter tool * pylint *. Also with * pylintrc *.

Documentation

pytest.ini

◆ Number of projects used: 69 ◆ Tools used: pytest

• pytest * configuration file. Describe the options at runtime.

Documentation

.flake8

◆ Number of projects used: 55 ◆ Tools used: flake8

The configuration file of the familiar static analysis tool * flake8 *.

Documentation

conftest.py

◆ Number of projects used: 32 ◆ Tools used: pytest

A file for defining common fixtures and util functions in * pytest *. If you create it in a package, you can use it only under that package, so you can arrange it according to the scope you want to use.

mypy.ini

◆ Number of projects used: 28 ◆ Tools used: mypy

Configuration file for the static type checking tool * mypy *. .. There was also one * .mypy.ini *.

Documentation

.isort.cfg

◆ Number of projects used: 23 ◆ Tools used: isort

A configuration file for * isort *, a tool that sorts Python code imports according to rules.

Documentation

.bumpversion.cfg

◆ Number of projects used: 21 ◆ Tools used: bumpversion

bumpversion is a command line tool that appropriately increases and rewrites various versions defined in the source code.

• .bumpversion.cfg * is the configuration file. If * .bumpversion.cfg * does not exist, it seems to go to * setup.cfg *.

manage.py

◆ Number of projects used: 20 ◆ Tools used: django configuration script?

django I don't really understand, but it says something like this. zulip/manage.py

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "zproject.settings")
    from django.conf import settings
    from django.core.management import execute_from_command_line
    from django.core.management.base import CommandError

    from scripts.lib.zulip_tools import log_management_command

    log_management_command(" ".join(sys.argv), settings.MANAGEMENT_LOG_PATH)

    os.environ.setdefault("PYTHONSTARTUP", os.path.join(BASE_DIR, "scripts/lib/pythonrc.py"))
    if "--no-traceback" not in sys.argv and len(sys.argv) > 1:
        sys.argv.append("--traceback")
    try:
        execute_from_command_line(sys.argv)
    except CommandError as e:
        print(e, file=sys.stderr)
        sys.exit(1)

versioneer.py

◆ Number of projects used: 15 ◆ Tools used: versioneer

This is also a tool that automatically raises the version of the project.

.pyup.yml

◆ Number of projects used: 14 ◆ Tools used: pyup

It seems that it will update the Python dependency file of the project via the pull request of GitHub / GitLab. I'm not sure.

.pep8speaks.yml

◆ Number of projects used: 11 ◆ Tools used: pep8speaks

It's a "GitHub app that automatically checks the Python code style with a pull request".

.style.yapf

◆ Number of projects used: 9 ◆ Tools used: YAPF

Google's Python code formatter. It is an abbreviation for "Yet Another Python Formatter".

tasks.py

◆ Number of projects used: 9 ◆ Tools used: Invoke?

Reference: https://equal-001.hatenablog.com/entry/2016/03/31/113000

pyinvoke is a module that can put together various commands.

Quoted from the official documentation. If you add a @ task decorator to the function as shown below,

from invoke import task

@task
def clean(c, docs=False, bytecode=False, extra=''):
    patterns = ['build']
    if docs:
        patterns.append('docs/_build')
    if bytecode:
        patterns.append('**/*.pyc')
    if extra:
        patterns.append(extra)
    for pattern in patterns:
        c.run("rm -rf {}".format(pattern))

@task
def build(c, docs=False):
    c.run("python setup.py build")
    if docs:
        c.run("sphinx-build docs docs/_build")

It seems that it can be easily called as a task from the CLI

$ invoke clean build

.landscape.yml

◆ Number of projects used: 8 ◆ Tools used: Prospector?

Prospector is a tool to analyse Python code and output information about errors, potential problems, convention violations and complexity.

It seems to be a code analysis tool.

noxfile.py

◆ Number of projects used: 7 ◆ Tools used: Nox

tox-like library

Other

Configuration files for tools that are not exclusive to Python, such as CI / CD, Docker / Vagrant, and document generation.

.travis.yml

◆ Number of projects used: 386 ◆ Tools used: Travis CI

CI。

Dockerfile

◆ Number of projects used: 111 ◆ Tools used: Docker

dockerfile.

.readthedocs.yml

◆ Number of projects used: 90 ◆ Tools used: readthedocs

Document generation tool Do you have * .readthedocs.yml * and * readthedocs.yml *?

It seems that it is used in quite a lot of projects, and many of the tools I have written so far are actually making documents with this.

.pre-commit-config.yaml ◆ Number of projects used: 88 ◆ Tools used: pre-commit

• pre-commit * is [Git hook](https://git-scm.com/book/ja/v2/Git-%E3%81%AE%E3%82%AB%E3%82%B9%E3% 82% BF% E3% 83% 9E% E3% 82% A4% E3% 82% BA-Git-% E3% 83% 95% E3% 83% 83% E3% 82% AF) * pre-commit * A tool for managing
• pre-commit * itself is made by Python and can be used in projects other than Python

.editorconfig

◆ Number of projects used: 83 ◆ Tools used: editorconfig

A tool for formatting code in various editors and IDEs. It can be used in other than Python.

.appveyor.yml

◆ Number of projects used: 79 ◆ Tools used: appveyor

CI / CD service.

.dockerignore

◆ Number of projects used: 76 ◆ Tools used: Docker

File definitions to ignore in Docker

mkdocs.yml

◆ Number of projects used: 34 ◆ Tools used: mkdocs

It's a fast, simple and luxurious static site generator for building project documents.

docker-compose.yml

◆ Number of projects used: 33 ◆ Tools used: docker-compose

docker-compose.

package.json

◆ Number of projects used: 32 ◆ Tools used: npm

npm package file.

In projects made with Python and JavaScript such as frappe / frappe, package management on the js side is probably done with * package.json *. ..

azure-pipelines.yml

◆ Number of projects used: 28 ◆ Tools used: Azure Pipelines

.deepsource.toml

◆ Number of projects used: 14 ◆ Tools used: deepsource

Source code analysis? It seems that it can be used other than Python

environment.yml

◆ Number of projects used: 13 ◆ Tools used: conda?

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file

Vagrantfile

◆ Number of projects used: 12 ◆ Tools used: Vagrant

package-lock.json

◆ Number of projects used: 12 ◆ Tools used: npm

npm lock file. It is managed together with package.json.

_config.yml

◆ Number of projects used: 11 ◆ Tools used: jekyll

jekyll is a Ruby tool for generating static sites. For example, this becomes like this?

Jenkinsfile

◆ Number of projects used: 10 ◆ Tools used: Jenkins

.eslintrc

◆ Number of projects used: 16 ◆ Tools used: Eslint

It shouldn't be related to Python. Is it for JS?

• .eslintrc.js * * .eslintrc.json * * .eslintrc.yml * etc.

Procfile

◆ Number of projects used: 10 ◆ Tools used: Heroku

A file that specifies the commands that your app executes when you start it on Heroku.

.codeclimate.yml

◆ Number of projects used: 10 ◆ Tools used: CodeClimate?

Code quality check, coverage measurement, etc. It can be used in other than Python.

yarn.lock

◆ Number of projects used: 9 ◆ Tools used: yarn

Yarn lock file. It is managed together with * package.json *.

runtime.txt

◆ Number of projects used: 7 ◆ Tools used: Heroku

Configuration file that specifies the runtime when deploying to Heroku

At the end

I got them in descending order of stars, so I wonder if there are few projects using modern tools? I had the impression that. There seems to be a lot to learn from well-known repositories such as Python project structure and test release scripts, so if you are interested, it may be fun to check it out. Please let us know if there are other commonly used tools / recommended tools.