The story that a hash error came out when using Pipenv

What is this

One day when I was using Pipenv, I got a mysterious error.

I had a hard time figuring out the cause, so I decided to write it as a memo. Eventually it was resolved, so if you are suffering from similar symptoms and just want to know how to deal with it, please go to Solution.

One day when I was using Pipenv, I noticed a mysterious error

The first time I noticed this error was when I wanted to set up an automated test in CI. I was writing code to run automated tests inside docker, but I noticed the following error when running pipenv sync or pipenv install inside docker.

An error occurred while installing readme-renderer==24.0 --hash=sha256:bb16f55b259f27f75f640acf5e00cf897845a8b3e4731b5c1a436e4b8529202f --hash=sha256:c8532b79afc0375a85f10433eca157d6b50f7d6990f337fa498c96cd4bfc203d! Will try again.
An error occurred while installing requests==2.22.0 --hash=sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4 --hash=sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31 --hash=sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4 --hash=sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31! Will try again.
An error occurred while installing requests-toolbelt==0.9.1 --hash=sha256:380606e1d10dc85c3bd47bf5a6095f815ec007be7a8b69c878507068df059e6f --hash=sha256:968089d4584ad4ad7c171454f0a5c6dac23971e9472521ea3b6d49d610aa6fc0! Will try again.
An error occurred while installing secretstorage==3.1.1 ; sys_platform == 'linux' --hash=sha256:20c797ae48a4419f66f8d28fc221623f11fc45b6828f96bdb1ad9990acb59f92 --hash=sha256:7a119fb52a88e398dbb22a4b3eb39b779bfbace7e4153b7bc6e5954d86282a8a! Will try again.
An error occurred while installing sphinx==2.2.1 --hash=sha256:31088dfb95359384b1005619827eaee3056243798c62724fd3fa4b84ee4d71bd --hash=sha256:52286a0b9d7caa31efee301ec4300dbdab23c3b05da1c9024b4e84896fb73d79! Will try again.
An error occurred while installing sphinx-git==11.0.0 --hash=sha256:6bf9d837de108c79fb7db585ebd590fd48f4d1f830b540420d0ca675f3b9f800! Will try again.
An error occurred while installing twine==3.1.0 --hash=sha256:1a87ae3f1e29a87a8ac174809bf0aa996085a0368fe500402196bda94b23aab3 --hash=sha256:ba8ba1b39987f1c22d9162f7dd3b4668388640e5f7158c15226624f88e464836! Will try again.

After trying various things, I found that running pipenv install or pipenv sync without a venv environment causes an error. Since the hash is displayed, I deleted Pipfile.lock for the time being, and then executed pipenv install`, but I still get an error. The situation confirmed at this time is as follows.

Well, what is this ...

Take a look at the Pipfile

Even after deleting both the venv environment and Pipfile.lock, an error still appears, so it is almost certain that the cause is Pipfile. The Pipfile at this time is as follows.

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
flake8 = "*" 
autopep8 = "*" 
isort = "*" 
autoflake = "*" 
pytest = "*" 
pytest-cov = "*" 
qiitacli = {path = "."}
twine = "*" 
sphinx = "*" 
sphinx-git = "*" 
pypandoc = "*" 

[packages]
click = ">=7.0"
qiita_v2 = ">=0.2.1"

[requires]
python_version = "3" 

[scripts]
main = "python qiitacli/client.py"

I don't know what's wrong with my brain ...

Take a look at the log

I tried to look at the output with the -v option when pipenv install. I was surprised that there was more output than I expected.

Here is the log that seems to be the cause.

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    tqdm>=4.14 from https://files.pythonhosted.org/packages/bb/62/6f823501b3bf2bac242bd3c320b592ad1516b3081d82c77c1d813f076856/tqdm-4.39.0-py2.py3-none-any.whl#sha256=5a1f3d58f3eb53264387394387fe23df469d2a3fab98c9e7f99d5c146c119873 (from twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1))    readme-renderer>=21.0 from https://files.pythonhosted.org/packages/c3/7e/d1aae793900f36b097cbfcc5e70eef82b5b56423a6c52a36dce51fedd8f0/readme_renderer-24.0-py2.py3-none-any.whl#sha256=c8532b79afc0375a85f10433eca157d6b50f7d6990f337fa498c96cd4bfc203d (from twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1))
    requests-toolbelt!=0.9.0,>=0.8.0 from https://files.pythonhosted.org/packages/60/ef/7681134338fc097acef8d9b2f8abe0458e4d87559c689a8c306d0957ece5/requests_toolbelt-0.9.1-py2.py3-none-any.whl#sha256=380606e1d10dc85c3bd47bf5a6095f815ec007be7a8b69c878507068df059e6f (from twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1))
    secretstorage; sys_platform == "linux" from https://files.pythonhosted.org/packages/82/59/cb226752e20d83598d7fdcabd7819570b0329a61db07cfbdd21b2ef546e3/SecretStorage-3.1.1-py3-none-any.whl#sha256=7a119fb52a88e398dbb22a4b3eb39b779bfbace7e4153b7bc6e5954d86282a8a (from keyring>=15.1->twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1))
Exception information:
Traceback (most recent call last):
  File "/hoge/.venv/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
    status = self.run(options, args)
  File "/hoge/.venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 382, in run
    resolver.resolve(requirement_set)
  File "/hoge/.venv/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 208, in resolve
    raise hash_errors
pip._internal.exceptions.HashErrors: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    tqdm>=4.14 from https://files.pythonhosted.org/packages/bb/62/6f823501b3bf2bac242bd3c320b592ad1516b3081d82c77c1d813f076856/tqdm-4.39.0-py2.py3-none-any.whl#sha256=5a1f3d58f3eb53264387394387fe23df469d2a3fab98c9e7f99d5c146c119873 (from twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1))    readme-renderer>=21.0 from https://files.pythonhosted.org/packages/c3/7e/d1aae793900f36b097cbfcc5e70eef82b5b56423a6c52a36dce51fedd8f0/readme_renderer-24.0-py2.py3-none-any.whl#sha256=c8532b79afc0375a85f10433eca157d6b50f7d6990f337fa498c96cd4bfc203d (from twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1))
    requests-toolbelt!=0.9.0,>=0.8.0 from https://files.pythonhosted.org/packages/60/ef/7681134338fc097acef8d9b2f8abe0458e4d87559c689a8c306d0957ece5/requests_toolbelt-0.9.1-py2.py3-none-any.whl#sha256=380606e1d10dc85c3bd47bf5a6095f815ec007be7a8b69c878507068df059e6f (from twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1))
    secretstorage; sys_platform == "linux" from https://files.pythonhosted.org/packages/82/59/cb226752e20d83598d7fdcabd7819570b0329a61db07cfbdd21b2ef546e3/SecretStorage-3.1.1-py3-none-any.whl#sha256=7a119fb52a88e398dbb22a4b3eb39b779bfbace7e4153b7bc6e5954d86282a8a (from keyring>=15.1->twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1))

You have added the --require-hashes option when you do pip install internally. It seems to be hash checking mode, and I didn't know its existence until I saw this log.

ʻIn --require-hashes mode, all requirements must have their versions pinned with ==. These do notIn summary, in hash checking mode, you have to specify all module versions with==`! !! It sounds like that. (English Wakaranai)

In the first place, the version of the module with no error is not specified in Pipfile, and the mystery only deepens.

Take a look at requirements.txt

Looking at the log, it seems that the Pipenv command generates a file equivalent to requirements.txt under / tmp and executes it with the -r option added to the pip command. I actually looked at the contents of those files.

As a method, just hit sudo cp -rf /tmp/pipenv*. repeatedly when executing Pipenv. There must be a better way, but for now.

Looking at one of the copied files, it looks like this

twine==3.1.0 --hash=sha256:1a87ae3f1e29a87a8ac174809bf0aa996085a0368fe500402196bda94b23aab3 --hash=sha256:ba8ba1b39987f1c22d9162f7dd3b4668388640e5f7158c15226624f88e464836

Hmmm, I'm generating requirements.txt with a hash value for each module and passing it to the -r option.

I checked the hash value, but it was the same as the file that was actually published.

I will try to install it in a venv environment created by hand using this requirements.txt file.

$ echo -n ' twine==3.1.0 --hash=sha256:1a87ae3f1e29a87a8ac174809bf0aa996085a0368fe500402196bda94b23aab3 --hash=sha256:ba8ba1b39987f1c22d9162f7dd3b4668388640e5f7158c15226624f88e464836' > requirements.txt
$ python3 -m venv venv
$ ./venv/bin/pip install -r requirements.txt --require-hashes
Collecting twine==3.1.0
  Using cached https://files.pythonhosted.org/packages/74/45/1016cad7eb7cbda959a701d1dfa88b9118306677018ac3de224a6a6e7751/twine-3.1.0-py3-none-any.whl
Collecting readme-renderer>=21.0
Collecting requests-toolbelt!=0.9.0,>=0.8.0
Collecting tqdm>=4.14
Collecting requests>=2.20
Collecting keyring>=15.1
Collecting importlib-metadata; python_version < "3.8"
Requirement already satisfied: setuptools>=0.7.0 in ./venv/lib/python3.6/site-packages (from twine==3.1.0->-r requirements.txt (line 1)) (42.0.1)
Collecting pkginfo>=1.4.2
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    readme-renderer>=21.0 from https://files.pythonhosted.org/packages/c3/7e/d1aae793900f36b097cbfcc5e70eef82b5b56423a6c52a36dce51fedd8f0/readme_renderer-24.0-py2.py3-none-any.whl#sha256=c8532b79afc0375a85f10433eca157d6b50f7d6990f337fa498c96cd4bfc203d (from twine==3.1.0->-r requirements.txt (line 1))
    requests-toolbelt!=0.9.0,>=0.8.0 from https://files.pythonhosted.org/packages/60/ef/7681134338fc097acef8d9b2f8abe0458e4d87559c689a8c306d0957ece5/requests_toolbelt-0.9.1-py2.py3-none-any.whl#sha256=380606e1d10dc85c3bd47bf5a6095f815ec007be7a8b69c878507068df059e6f (from twine==3.1.0->-r requirements.txt (line 1))
    tqdm>=4.14 from https://files.pythonhosted.org/packages/bb/62/6f823501b3bf2bac242bd3c320b592ad1516b3081d82c77c1d813f076856/tqdm-4.39.0-py2.py3-none-any.whl#sha256=5a1f3d58f3eb53264387394387fe23df469d2a3fab98c9e7f99d5c146c119873 (from twine==3.1.0->-r requirements.txt (line 1))
    requests>=2.20 from https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl#sha256=9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31 (from twine==3.1.0->-r requirements.txt (line 1))
    keyring>=15.1 from https://files.pythonhosted.org/packages/b1/08/ad1ae7262c8146bee3be360cc766d0261037a90b44872b080a53aaed4e84/keyring-19.2.0-py2.py3-none-any.whl#sha256=f5bb20ea6c57c2360daf0c591931c9ea0d7660a8d9e32ca84d63273f131ea605 (from twine==3.1.0->-r requirements.txt (line 1))
    importlib-metadata; python_version < "3.8" from https://files.pythonhosted.org/packages/f6/d2/40b3fa882147719744e6aa50ac39cf7a22a913cbcba86a0371176c425a3b/importlib_metadata-0.23-py2.py3-none-any.whl#sha256=d5f18a79777f3aa179c145737780282e27b508fc8fd688cb17c7a813e8bd39af (from twine==3.1.0->-r requirements.txt (line 1))
    pkginfo>=1.4.2 from https://files.pythonhosted.org/packages/e6/d5/451b913307b478c49eb29084916639dc53a88489b993530fed0a66bab8b9/pkginfo-1.5.0.1-py2.py3-none-any.whl#sha256=a6d9e40ca61ad3ebd0b72fbadd4fba16e4c0e4df0428c041e01e06eb6ee71f32 (from twine==3.1.0->-r requirements.txt (line 1))

Oh! I reproduced it!

But the error is in a module different from the module described in requirements.txt ...

In this hash checking mode, it is necessary to describe all dependent packages by specifying the version. But why don't you get an error when you normally do pipenv install?

What is --no-deps

After checking the log for a while, I found that it has an option --no-deps when installing with pip when it is working properly.

$ ['/hoge/.venv/bin/pip', 'install', '--verbose', '--upgrade', '-r', '/tmp/pipenv-9owwlzhi-requirements/pipenv-65yqxf03-requirement.txt', '-i', 'https://pypi.org/simple', '--require-hashes']
$ ['/hoge/.venv/bin/pip', 'install', '--verbose', '--upgrade', '--no-deps', '-r', '/tmp/pipenv-5dvxlnyt-requirements/pipenv-pvatvj5c-requirement.txt', '-i', 'https://pypi.org/simple', '--require-hashes']

Oh, it looks like this!

--no-deps is Don't install package dependencies. That is, it is an option that does not install dependent packages. Naruhode If this is the case, the requirements.txt mentioned earlier seems to work normally.

$ ./venv/bin/pip install -r requirements.txt --require-hashes --no-deps 
Collecting twine==3.1.0
  Using cached https://files.pythonhosted.org/packages/74/45/1016cad7eb7cbda959a701d1dfa88b9118306677018ac3de224a6a6e7751/twine-3.1.0-py3-none-any.whl
Installing collected packages: twine
Successfully installed twine-3.1.0

did it.

The cause has been grasped little by little.

The only remaining mystery is ** Why didn't you add the --no-deps option **?

The last key is editable

At the end, you have to read the source of Pipenv.

https://github.com/pypa/pipenv

Let's see where the pip install ... command is actually executed when the pipenv install is executed.

The place where the pip command is actually executed was the function pip_install in pipenv / core.py. There were various call orders such as do_install-> do_init-> do_install_dependencies-> batch_install-> pip_install, but ... In conclusion, the following part of the batch_install function was flagged as to whether or not to add the --no-deps option.

        if not PIPENV_RESOLVE_VCS and is_artifact and not dep.editable:
            install_deps = True 
            no_deps = False

dep is a class that contains information about the modules to install. Not PIPENV_RESOLVE_VCS, but ʻis_artifact, and the module is not ʻeditable ...

I don't understand at all

Somehow, "modules installed from source that are not flagged as editable" is related to the granting of --no-deps. After pipenv install -h, there was something like this.

$ pipenv install -h
Abbreviation
  -e, --editable TEXT      An editable python package URL or path, often to a
                           VCS repo.

Ah,,,

Solution

Let's look again at the Pipfile that caused the error.

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
flake8 = "*"
autopep8 = "*"
isort = "*"
autoflake = "*"
pytest = "*"
pytest-cov = "*"
qiitacli = {path = "."}
twine = "*"
sphinx = "*"
sphinx-git = "*"
pypandoc = "*"

[packages]
click = ">=7.0"
qiita_v2 = ">=0.2.1"

[requires]
python_version = "3"

[scripts]
main = "python qiitacli/client.py"

Yes the cause is

qiitacli = {path = "."}

It was this guy.

I used to do pipenv install . when installing a module under development locally, but this was a problem. The correct answer is pipenv install -e .. The resulting Pipfile looks like this.

qiitacli = {editable = true,path = "."}

It has an editable flag.

In this state, the installation is completed normally.

Summary

In the past, I used to install it with the -e option, but after I realized that I could install it without the option, I decided to install it as it was. (Perhaps)

By the way, it was written firmly in the official document. https://pipenv-ja.readthedocs.io/ja/translate-ja/basics.html#editable-dependencies-e-g-e

Let's at least understand the meaning of the options we are using. myself.

Recommended Posts

The story that a hash error came out when using Pipenv
A story that was convenient when I tried using the python ip address module
A story that stumbled when using pip in a proxy environment
[Linux / GCP] Corrected the error that occurred when using the Git command.
A story when a directory is buggy with the django-admin start project command in a virtual environment using Pipenv
A addictive story when using tensorflow on Android
Generate a hash value using the HMAC method.
It is a piggybacking story about the service that returns "Nyan" when you ping
A story that failed when trying to remove the suffix from the string with rstrip
A story that got stuck when trying to upgrade the Python version on GCE
A story that struggled with the common set HTTP_PROXY = ~
SoC FPGA: A small story when using on Linux
A story that an error occurred when PyInstaller was used in a program that uses googleapiclient
There is a pattern that the program did not stop when using Python threading
A story that analyzed the delivery of Nico Nama.
A swampy story when using firebase on AWS lamda
A story that makes it easy to estimate the living area using Elasticsearch and Python
A story that stumbled when I made a chatbot with Transformer
I got a Value Error when using JUMAN ++ with PyKNP
A story about installing matplotlib using pip with an error
The story of creating a database using the Google Analytics API
When using property, use a class that inherits object (new-style class)
A story that didn't work when I tried to log in with the Python requests module
When incrementing the value of a key that does not exist
The story that the Homebrew environment was blown away when Anaconda was installed
A story that I fixed when I got Lambda logs from Cloudwatch Logs
A story that struggled to handle the Python package of PocketSphinx
Precautions when using a list or dictionary as the default argument
A story about a build error in a shared library that references libusb
[Ev3dev] Create a program that captures the LCD (screen) using python
The story of creating a site that lists the release dates of books
The story that the guard was confined when the laboratory was converted to IoT
The story of making a module that skips mail with python
An error that does not work as expected when calling the tkinter module in a text editor
The story of making a slackbot that outputs as gif or png when you send the processing code
A memorandum when using beautiful soup
Precautions when using the urllib.parse.quote function
A story about using Python's reduce
The story of writing a program
A story that visualizes the present of Qiita with Qiita API + Elasticsearch + Kibana
When you make a mistake in the directory where you execute `pipenv install`
The story of developing a web application that automatically generates catchphrases [MeCab]
Estimate the probability that a coin will appear on the table using MCMC
The story of making a package that speeds up the operation of Juman (Juman ++) & KNP
A useful note when using Python for the first time in a while
A story that suffered from OS differences when trying to implement a dissertation