Recommendation of building a portable Python environment with conda

Overview

Recently, it seems that Anaconda is often used to quickly build a Python environment for data analysis.

For Anaconda and its minimal configuration Miniconda, use the included package manager conda. You can use it to manage the entire environment, including Python itself. In conventional Python development, it was necessary to build an environment using individual tools according to the purpose as follows.

With Anaconda / Miniconda, these can be done only with conda.

In situations such as team development or running deliverables on a production server, it is useful to be able to quickly rebuild the built environment on another machine. When using pip and pyenv, the version of the library and Python itself was managed by preparing files such as requirements.txt and .python-version, respectively. With conda, you can easily rebuild your environment from there by exporting your preferences in YAML format.

The detailed method is explained below.

Miniconda installation

Anaconda is an all-in-one platform that includes many major packages, but if you want to build the minimum required environment for each project like this time, it is smarter to use Miniconda.

To install Miniconda, start the downloaded installer as described in here.

$ bash Miniconda2-latest-MacOSX-x86_64.sh

For Mac, you can also install it with brew cask.

$ brew install Caskroom/cask/miniconda

In the following, it is assumed that Miniconda is already installed and each command including conda is in the PATH.

Building the environment

Here, let's create a Python 3.5 environment with the name myenv. Since conda treats Python itself and packages equally as components of the environment, packages can be installed at the same time.

$ conda create --name myenv python=3.5 numpy=1.11.1

Activate the environment you created.

$ source activate myenv

You can also install additional packages.

$ conda install scipy

Not all packages registered with PyPI can be installed with conda. Packages that cannot be installed with conda can still be installed with pip. pip is included in the environment created by conda from the beginning.

$ pip install peewee

Export and reuse preferences

You can export your environment settings in YAML format by running conda env export with your environment enabled.

$ conda env export > myenv.yaml

The exported file looks like this: Packages installed with pip are also exported correctly.

myenv.yaml


name: myenv
dependencies:
- mkl=11.3.3=0
- numpy=1.11.1=py35_0
- openssl=1.0.2h=1
- pip=8.1.2=py35_0
- python=3.5.1=5
- readline=6.2=2
- scipy=0.17.1=np111py35_1
- setuptools=23.0.0=py35_0
- sqlite=3.13.0=0
- tk=8.5.18=0
- wheel=0.29.0=py35_0
- xz=5.2.2=0
- zlib=1.2.8=3
- pip:
  - peewee==2.8.1

With this file, you can easily rebuild the same environment on another machine.

$ conda env create --file myenv.yaml

Write the configuration file yourself

The configuration file exported by export will also include packages installed due to the dependencies of the intentionally installed packages. If you want to specify only the libraries that are used directly as the structure of the project, you need to write the configuration file yourself.

myenv.yaml


name: myenv
dependencies:
- python=3.5.1
- numpy=1.11.1
- scipy=0.17.1
- pip:
  - peewee==2.8.1

The recommendation is to follow Gemfile, Gemfile.lock in Ruby's Bundler, describe only the library to be used directly in myenv.yaml, and export the environment created using this. The method is to save it with a name like myenv.frozen.yaml. That way, you can look at myenv.yaml to see which packages are used directly in your project, and use myenv.frozen.yaml to reconstruct the exact same environment, including dependent packages.

reference

Recommended Posts

Recommendation of building a portable Python environment with conda
Building a virtual environment with Python 3
conda memorandum: Building a Python environment with supercomputer ITO
[Pyenv] Building a python environment with ubuntu 16.04
Building a Python3 environment with Amazon Linux2
Building a Python 3.6 environment with Windows + PowerShell
Building a python environment with virtualenv and direnv
Building a Python environment with WLS2 + Anaconda + PyCharm
Create a virtual environment with conda in Python
Building a Python virtual environment
Building a Python virtual environment
Building a kubernetes environment with ansible 2
Building a Python environment on Mac
Building a Python environment on Ubuntu
Create a virtual environment with Python!
Building a kubernetes environment with ansible 1
[Note] List of basic commands for building python / conda environment
Building a Windows 7 environment for getting started with machine learning with Python
[Python] Create a virtual environment with Anaconda
[Mac] Building a virtual environment for Python
Building a conda environment for ROS users
Recommendation of Altair! Data visualization with Python
Build a python virtual environment with pyenv
Introduce graphviz into a portable python environment
Build a modern Python environment with Neovim
Building a Python development environment for AI development
Get a list of packages installed in your current environment with python
[AtCoder] Solve A problem of ABC101 ~ 169 with Python
Prepare the execution environment of Python3 with Docker
Build a python environment with ansible on centos6
Building an Anaconda environment for Python with pyenv
[Python] Build a Django development environment with Docker
Use multiple versions of python environment with pyenv
Create a python3 build environment with Sublime Text3
Solve A ~ D of yuki coder 247 with python
[Python] Web development preparation (building a virtual environment)
Poetry-virtualenv environment construction with python of centos-sclo-rh ~ Notes
A memo when creating a python environment with miniconda
Think about building a Python 3 environment in a Mac environment
Commands for creating a python3 environment with virtualenv
Work in a virtual environment with Python virtualenv.
Flow of creating a virtual environment with Anaconda
Build a Python environment with OSX El capitan
Quickly build a Python Django environment with IntelliJ
Building a Python environment on a Sakura VPS server
Build a Python machine learning environment with a container
Build a python execution environment with VS Code
Get a quick Python development environment with Poetry
Unification of Python environment
Python environment with docker-compose
Create a Python environment
Virtual environment with Python 3.6
A memo connected to HiveServer2 of EMR with python
Building a Python environment for pyenv, pyenv-virtualenv, Anaconda (Miniconda)
Write about building a Python environment for writing Qiita Qiita
Building a Docker working environment for R and Python
Build a python virtual environment with virtualenv and virtualenvwrapper
Build a python environment for each directory with pyenv-virtualenv
Create a python development environment with vagrant + ansible + fabric
I made a Python3 environment on Ubuntu with direnv.
Building an environment for natural language processing with Python