When I did conda install hogehoge
, an unplanned package (or python
) was updated and the whole thing died, so I'll write a memorandum.
What I wanted to do: I wanted to create my own image by adding some libraries based on the docker image of pytorch
.
What I did: I wrote conda install
obediently in Dockerfile
What happened: The version of python
changed during conda install
, and the conda environment was damaged and could not be used properly.
Genin: Mismatch between conda
and python
versions
What I had to do (Conclusion): Write what you want to fix in a file called pinned
(this time solved withpython == 3.6. *
)
(Or include the conda
that matches the updated version of python
; not done this time; at the bottom of the reference link)
Is it possible to lock versions of packages in Anaconda? - Stack Overflow How to pin Conda | Damian's blog Notes on changing the python version of conda --Qiita
I wrote the following Dockerfile
(the old image I'm using may be part of the problem ...).
FROM pytorch/pytorch:0.4.1-cuda9-cudnn7-devel
RUN apt update && apt upgrade -y
#This will be a problem later
# joblib etc
RUN conda install -y \
joblib \
matplotlib \
scikit-learn \
tqdm
Then build.
$ docker build -t mypytorch .
At this time, conda install
displayed these as packages to be updated (the part that will be a problem later).
The following packages will be UPDATED:
ca-certificates: 2018.03.07-0 --> 2020.1.1-0
certifi: 2018.4.16-py36_0 --> 2019.11.28-py38_0
freetype: 2.9.1-h8a8886c_0 --> 2.9.1-h8a8886c_1
libedit: 3.1.20170329-h6b74fdf_2 --> 3.1.20181209-hc058e9b_0
libgcc-ng: 7.2.0-hdf63c60_3 --> 9.1.0-hdf63c60_0
libpng: 1.6.34-hb9fc6fc_0 --> 1.6.37-hbc83047_0
libstdcxx-ng: 7.2.0-hdf63c60_3 --> 9.1.0-hdf63c60_0
mkl: 2018.0.3-1 --> 2020.0-166
mkl_fft: 1.0.4-py36h4414c95_0 --> 1.0.15-py38ha843d7b_0
mkl_random: 1.0.1-py36h4414c95_1 --> 1.1.0-py38h962f231_0
ncurses: 6.1-hf484d3e_0 --> 6.2-he6710b0_0
numpy: 1.14.5-py36h1b885b7_4 --> 1.18.1-py38h4f9e942_0
numpy-base: 1.14.5-py36hdbf6ddf_4 --> 1.18.1-py38hde5b4d6_1
openssl: 1.0.2o-h20670df_0 --> 1.1.1d-h7b6447c_4
pip: 10.0.1-py36_0 --> 20.0.2-py38_1
python: 3.6.5-hc3d631a_2 --> 3.8.1-h0371630_1
readline: 7.0-ha6073c6_4 --> 7.0-h7b6447c_5
scipy: 1.1.0-py36hc49cb51_0 --> 1.4.1-py38h0b6359f_0
setuptools: 39.2.0-py36_0 --> 45.2.0-py38_0
six: 1.11.0-py36h372c433_1 --> 1.14.0-py38_0
sqlite: 3.23.1-he433501_0 --> 3.31.1-h7b6447c_0
tk: 8.6.7-hc745277_3 --> 8.6.8-hbc83047_0
wheel: 0.31.1-py36_0 --> 0.34.2-py38_0
zlib: 1.2.11-ha838bed_2 --> 1.2.11-h7b6447c_3
Now that the build is complete, we will triumphantly make a container.
$ docker run -it --rm mypytorch bash
When I try to use conda
type commands in the container, it doesn't work.
root@ce88f6b1ba73:/workspace# conda
Traceback (most recent call last):
File "/opt/conda/bin/conda", line 7, in <module>
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Others ʻipythondoesn't work, or it can't be helped, so even if you try from
python` as below, it won't work.
root@ce88f6b1ba73:/workspace# python
Python 3.8.1 (default, Jan 8 2020, 22:29:32)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
>>>
So, I have created an image of something that I want to move but nothing that moves.
This was the problem. The version of python
has been updated to 3.8.1.
The following packages will be UPDATED:
...
python: 3.6.5-hc3d631a_2 --> 3.8.1-h0371630_1
...
It seems that it will not work if only python
is updated without changing the version of conda
. There are two ways to solve it.
conda
and python
togetherpython
This time, we will fix the version. To fix the version, just write what you want to fix in conda-meta / pinned
under the management directory of anaconda / miniconda (how it will be handled when multiple environments are created has not been verified).
This time, since python
is fixed to 3.6 series, it will be as follows (here, the installation destination is / opt
. This depends on your environment).
$ echo "python=3.6.*" >> /opt/conda/conda-meta/pinned
It will work if this is in the Dockerfile. In other words, it becomes as follows.
FROM pytorch/pytorch:0.4.1-cuda9-cudnn7-devel
RUN apt update && apt upgrade -y
#Fixed version before conda install
RUN echo "python=3.6.*" >> /opt/conda/conda-meta/pinned
# joblib etc
RUN conda install -y \
joblib \
matplotlib \
scikit-learn \
tqdm
I don't think it's an accident that happens very often, but I hope it helps someone.
Recommended Posts