conda memorandum: Building a Python environment with supercomputer ITO

It is a memorandum of conda. In the login node (bash) of Supercomputer ITO, [Miniconda](https://docs.conda.io/ En / latest / miniconda.html) to build a local environment. We have confirmed the operation with the login node of subsystem A of the supercomputer ITO and the basic front end (virtual), but I think that it is generally applicable to the Linux (bash) environment. We will also actively utilize Intel's Distribution for Python. ** Addendum (2020.4.11): I encountered a problem with Intel Distribution for Python, so I stopped actively using it. ** **

Install Miniconda

Install Miniconda in your local environment. Anaconda is fine, but it's huge, so version conflicts are likely to occur and handling is quite difficult. It is recommended to create and operate a virtual environment for each job with Miniconda. Log in to the login node and change to the appropriate directory for downloading files (for example, ~ / Downloads /). Get Miniconda with wget and install it.

$ wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ bash Miniconda3-latest-Linux-x86_64.sh

Specify the installation directory with full path if necessary, and answer yes for everything else. If you answer yes to the last question, the settings will be added to the environment setting file (~ / .bashrc). After the installation is complete, log in again or in your home directory, do the following:

$ source .bashrc

By doing this, you can use the installed Python environment immediately every time you log in. First, keep conda up to date.

$ conda update conda

From time to time, keep your conda up to date.

Uninstall Miniconda

Uninstalling Miniconda is easy, so if something goes wrong, uninstall it (https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html). However, it is recommended to install it again. Erase all directories where Miniconda is installed. If it is installed in the miniconda3 directory under the local directory under your home directory, do the following:

$ rm -rf ~/local/miniconda3

In addition, delete the related hidden files and directories (only those that exist) created in your home directory as follows.

$ rm -rf ~/.condarc ~/.conda

Finally, delete the following part of the .bashrc file that exists in your home directory. This is the part that will be added to .bashrc if you answer yes to the last question during the installation of Miniconda.

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/usr1/m00000a/local/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/home/usr1/m00000a/local/miniconda3/etc/profile.d/conda.sh" ]; then
        . "/home/usr1/m00000a/local/miniconda3/etc/profile.d/conda.sh"
    else
        export PATH="/home/usr1/m00000a/local/miniconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

About the package repository (channel)

The package repository (storage location) is called channel. Various institutions (individuals) provide packages. You can check it on Anaconda Cloud (package management service by Anaconda). For example, for TensorFlow, there are many channels as shown on this page (https://anaconda.org/search?q=tensorflow). There are typical channels, and it is normal to install packages from those typical channels. However, when it comes to specialized packages, you may need to install by specifying the channel. The channel in the official repository is defaults. Also, the community-based repository conda-forge manages more packages than the official channel and is often used.

operation of channel

We will introduce how to get a list of channels, how to add / delete channels, and how to change the priority of channels.

Get channel list conda config --get channels

Let's check the list of current channels. Immediately after installing Miniconda, there is a defaults channel as shown below.

$ conda config --get channels
--add channels 'defaults'   # lowest priority

First, add the often used conda-forge. I think the following state is the standard environment. Since conda-forge is highest priority (first), the package of conda-forge is preferentially installed.

$ conda config --add channels conda-forge

$ conda config --get channels
--add channels 'defaults'   # lowest priority
--add channels 'conda-forge'   # highest priority

In the following, we will introduce channel operations item by item.

Add channel to the beginning conda config --add channels channel_name

Add ʻintel` channel provided by Intel to the top (highest priority).

$ conda config --add channels intel

$ conda config --get channels
--add channels 'defaults'   # lowest priority
--add channels 'conda-forge'
--add channels 'intel'   # highest priority

Remove channel conda config --remove channels channel_name

Delete the intel channel.

$ conda config --remove channels intel

$ conda config --get channels
--add channels 'defaults'   # lowest priority
--add channels 'conda-forge'   # highest priority

Add channel to end conda config --append channels channel_name

Add intel channel to the end (lowest priority).

$ conda config --append channels intel

$ conda config --get channels
--add channels 'intel'   # lowest priority
--add channels 'defaults'
--add channels 'conda-forge'    # highest priority

Change the first channel conda config --add channels channel_name

Set the existing intel channel to the top (highest priority). To start it, use --add. To end it, use --append.

$ conda config --add channels intel
Warning: 'intel' already in 'channels' list, moving to the top

$ conda config --get channels
--add channels 'defaults'   # lowest priority
--add channels 'conda-forge'
--add channels 'intel'   # highest priority

Virtual environment

If you install the package in the default environment (base) of Miniconda, the dependency often fails and you may get to reinstall Miniconda. Therefore, it is a rule of thumb to prepare a virtual environment (v_env for short) for each purpose.

Building a virtual environment conda create -n v_env [python = version]

Build the virtual environment my_env.

$ conda create -n my_env

You can also specify the Python version. In the following, version 3.6 is specified.

$ conda create -n my_env python=3.6

To build the virtual environment my_intel_env using the Intel distribution, special operations are required as follows.

$ conda create -n my_intel_env intelpython3_core python=3

Changing the intelpython3_core part to intelpython3_full will build a complete Intel distribution. Also, if you change python = 3 to python = 2, the Python version will be 2. Note that Python 2 is not usually used.

Checking the virtual environment conda info -e

Check the available virtual environment and the selected environment. The execution result is the author's environment in the supercomputer ITO, and m00000a is the user home directory name (tentative name) of the supercomputer. In the environment where * is currently selected, it is base.

$ conda info -e
# conda environments:
#
base                  *  /home/usr1/m00000a/local/miniconda3
my_env                   /home/usr1/m00000a/local/miniconda3/envs/my_env
my_intel_env             /home/usr1/m00000a/local/miniconda3/envs/my_intel_env

Enter the virtual environment conda activate v_env

Enable (activate) the virtual environment my_intel_env and check the virtual environment.

$ conda activate my_intel_env

$ conda info -e
# conda environments:
#
base                     /home/usr1/m00000a/local/miniconda3
my_env                   /home/usr1/m00000a/local/miniconda3/envs/my_env
my_intel_env          *  /home/usr1/m00000a/local/miniconda3/envs/my_intel_env

You can see that * has moved and my_intel_env in the virtual environment has been activated.

Exiting the virtual environment conda deactivate

Execute conda deactivate to exit the virtual environment and return to base. If you execute conda info -e, you can see that * has returned to base.

$ conda deactivate

$ conda info -e
# conda environments:
#
base                  *  /home/usr1/m00000a/local/miniconda3
my_env                   /home/usr1/m00000a/local/miniconda3/envs/my_env
my_intel_env             /home/usr1/m00000a/local/miniconda3/envs/my_intel_env

Remove virtual environment conda remove -n v_env --all

To delete the virtual environment my_env, do the following.

$ conda remove -n my_env --all

Package management

Let's manage packages in the virtual environment my_intel_env.

Preparation

Currently, it is a base environment. First, update conda.

$ conda update conda

Next, activate the virtual environment my_intel_env where you want to install the package.

$ conda activate my_intel_env

Install package conda install package_list

You can also install multiple packages at once by arranging the package names separated by spaces. An example is shown below.

$ conda install jupyterlab matplotlib netCDF4 pandas xarray

Install package with channel conda install -c channel_name package_list

If you become more specialized, you may need to install a package that specifies a channel. Also, by specifying the channel, it becomes clear from which channel to install the package, which is recommended. To install tensorflow from anaconda channel:

$ conda install -c anaconda tensorflow

You can also specify a channel when building a virtual environment. The following is an example of building a virtual environment tf by specifying anaconda channel.

$ conda create -c anaconda -n tf

Checking installed packages conda list

To check the installed packages in the current environment, do the following.

$ conda list

If you give a virtual environment name to -n, you can check the installed packages in that virtual environment.

$ conda list -n my_env

Uninstall package conda uninstall package_list

To uninstall the package xarray, do the following. If you arrange multiple packages separated by spaces, you can uninstall them all at once.

$ conda uninstall xarray 

Write virtual environment settings conda env export -n v_env> file.yaml

If you write the information of the virtual environment to a file, you can easily build the same environment on other machines. Execute with base. Virtual environment Write the environment of my_intel_env to the my_intel_env.yaml file. This yaml file will be created in your current directory. The explanation of yaml is Wikipedia ([English version](https://en.wikipedia. org / wiki / YAML))).

$ conda env export -n my_intel_env > my_intel_env.yaml

However, it doesn't seem to work if you just created a virtual environment and no packages are installed.

Rebuild virtual environment conda env create -f file.yaml

To rebuild the environment on another machine using my_intel_env.yaml, do the following.

$ conda env create -f my_intel_env.yaml

Clone supercomputer vendor environment

The supercomputer ITO has an Intel Python environment installed by the vendor. If you use this as it is, the above installation work is not necessary, but you cannot install the package as it is. Therefore, it is necessary to clone the vendor environment as a local virtual environment. This method depends on the vendor environment and is not always well maintained and is not recommended. In fact, I ran into a bug where numpy couldn't be imported. The following is a summary for reference. This case is the information obtained from the usage consultation of the Information Infrastructure Research and Development Center of Kyushu University.

Clone the server's conda environment locally

Set the virtual environment name as intel2019up4 and build it as follows. It takes time to execute (it took about 7 minutes). The target that can be cloned depends on the maintenance status of the vendor.

$ source /home/app/intel/intel2019_up4/intelpython3/bin/activate
$ conda create -n intel2019up4 --clone="/home/app/intel/intel2019_up4/intelpython3"
$ source activate intel2019up4

Let's check.

$ conda info -e
# conda environments:
#
intel2019up4          *  /home/usr1/m00000a/.conda/envs/intel2019up4
root                     /home/app/intel/intel2019_up4/intelpython3

To get out of this environment, do the following.

$ source deactive intel2019up4