How to create a Conda package

How to create a Conda package

The user has Anaconda Python under his home directory If installed, Available in Conda package as well as Python Applications, libraries, etc. It will be easy to use.

This article describes how to create a Conda package in Anaconda Python. This is a document for Linux.

Conda package

As for Python packages, packages distributed by PyPI (Python Package Index) are known. These can be installed with the pip command.

** Installation example with pip command **

$ pip install shpinx

PyPI packages are created with a packaging tool called setuptools. Because it is a slightly quirky package creation method There are some difficulties in usability.

In Anaconda Python You can add / remove packages with the Conda command, An easy way to create a package is provided.

The Conda package is downloaded from the following location:

• defaults: defaults:
  • https://repo.continuum.io/pkgs/
• Anaconda Cloud (public donation repository):
  • https://anaconda.org/

Preparing to create a Conda package

If you just want to build a Conda package You can use miniconda.

• miniconda download site
  • https://repo.continuum.io/miniconda/

** How to install miniconda **

$ wget https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-x86_64.sh
$ bash Miniconda3-4.3.14-Linux-x86_64.sh -b -p $HOME/conda/x64
$ echo export PATH=$HOME/conda/x64/bin:$PATH >> $HOME/.bash_profile
$ source $HOME/.bash_profile

Next, in the root environment of Anaconda Python Install the conda-build package.

$ conda install -n root conda-build

It is recommended to use the latest version of conda-build. Even after installation, you can update as follows.

$ conda upgrade -n root conda
$ conda upgrade -n root conda-build

directory structure of conda-build

Miniconda3 if you have Miniconda3 installed under $ HOME / conda / x64 conda-build puts the files in the following directories:

$HOME/conda/x64/conda-bld/
├── git_cache           --Source downloaded with git
├── hg_cache            --Source downloaded with mercurial
├── linux-64            --conda package(x86_64:System dependent)
├── linux-ppc64le       --conda package(ppc64le:System dependent)
├── noarch              --conda package(noarch)
├── src_cache           --Source downloaded with curl
└── svn_cache           --Source downloaded with subversion

The package is built according to the recipe (more on that later), When you install the package, it is usually extracted to the following directory.

$HOME/conda/x64
├── bin
├── etc
├── lib
├── include
├── man
└── share

If you put it in a conda package, Under the user's home directory You can add / remove packages with user privileges.

Convert from existing package to Conda package

You can find it on CPAN, CRAN, luarocks.org, PyPI by using the conda skeleton command. You can convert an existing package to a Conda package.

• cpan -- Perl
  • CPAN(Perl Archive Netowkr / cpan.org)
• cran -- R
  • CRAN(Comprehensive R Archive Network / cran.r-project.org)
• luarocks -- LUA
  • luarocks.org
• pypi - Python
  • Python Packaging Index (PyPI / pypi.python.org)

For example, pyinstrument distributed by PyPI To make it a Conda package, execute the command as follows.

$ conda skeleton pypi autopep8
Using url https://pypi.python.org/packages/0a/fc/c541b2fa1b244e0484216076b95468dc011ae90016b8f37333a24a11e468/autopep8-1.3.2.tar.gz (105 KB) for autopep8.
Downloading autopep8
INFO:fetch.start:('autopep8-1.3.2', 107915)
INFO:fetch.update:16384
INFO:fetch.update:32768
INFO:fetch.update:49152
INFO:fetch.update:65536
INFO:fetch.update:81920
INFO:fetch.update:98304
INFO:fetch.update:107915
INFO:fetch.stop:None
Unpacking autopep8...
done
working in /tmp/tmptgwzsstsconda_skeleton_autopep8-1.3.2.tar.gz

The following NEW packages will be INSTALLED:

    openssl:    1.0.2l-0           
    pip:        9.0.1-py35_1       
    python:     3.5.3-1            
    pyyaml:     3.12-py35_0   hpcs 
    readline:   6.2-2         local
    setuptools: 27.2.0-py35_0      
    sqlite:     3.13.0-0           
    tk:         8.5.18-0           
    wheel:      0.29.0-py35_0      
    xz:         5.2.2-1            
    zlib:       1.2.8-3            

Applying patch: '/tmp/tmptgwzsstsconda_skeleton_autopep8-1.3.2.tar.gz/pypi-distutils.patch'
patching file core.py
Hunk #1 succeeded at 167 with fuzz 2 (offset 1 line).
Writing recipe for autopep8
INFO:conda_build.config:--dirty flag not specified.  Removing build folder after successful build/test.

When executed, it will be in the current directory A directory with the package name sphinx is created The required files will be created.

$ $ ls autopep8/
bld.bat  build.sh  meta.yaml

These files are called recipes for Conda packages.

If you give the conda build command the directory where the recipe is stored, You can create a Conda package.

conda build autopep8

Doing this will create a package file (actually a tarball).

/home/iisaka/conda/x64/conda-bld/linux-64/autopep8-1.3.2-py35_0.tar.bz2

Conda package naming convention

The Conda package creates a package file with the same naming convention as it does now.

package name-version-Build type_Build number.tar.bz2

• Build type --py26: Package for Python 2.6 --py27: Package for Python 2.7 --py33: Package for Python 3.3 --py34: Package for Python 3.4 --py35: Package for Python 3.5 --noarch: Architecture-independent package --None: Binary package
• Build number --Arbitrary number: Usually starts at 0 and increases with each release.

Conda packages for different Python versions

When you install Miniconda3-4.3.14 The default Python version is 3.6. On this platform If you want to create a Conda package for Python 2.7, Build as follows.

$ conda build --python=2.7 autopep8

The file created as a result of the build looks like this:

/home/iisaka//conda/x64/conda-bld/linux-64/autopep8-1.3.2-py27_0.tar.bz2

Channel registration

Channel registration

For example, on a Linux machine with x86_64 architecture With conda installed under / home / iisaka / py3 When you build the package, it will be saved in the following directory.

• '/home/iisaka/conda/x64/conda-bld/linux-64'
• '/home/iisaka/conda/x64/conda-bld/noarch'

Let's register this as a channel.

$ conda config --add channels file:///home/iisaka/conda/x64/conda-bld

recipe

The recipe consists of the following files.

• meta.yaml --Package metadata
• build.sh --build script
• bld.bat --Build script on Windows
• run_test. {py, pl, sh, bat} --test script (optional)
• • .patch --patch file

Package metadata

Package metadata Package name, package number, build number, how to get the source, etc. The file is written in YAML format and the file name is meta.yaml.

#Comments from the pound sign to the end of the line
package:
  name:package name
  version:Version number

source:
  fn:File name to save
  url:URL to download the source
#The checksum value can be any one, or omitted.
  md5:Checksum value of md5sum
  sha1:checksum value of sha1sum
  sha256:checksum value for sha256sum

#   patches:    #List of patch files(Optional)
#      - fix1.patch
#      - fix2.patch
build:
  number: 0       #Build number

requirements:     #Describe the dependent package name
  build:          #Packages required at build time
    - python
    - setuptools

  run:            #Packages required at run time
    - python

about:      #Describe the package description
  home:     #URL of the site or project that distributes the source
  license:  #List license GPL2, GPL3, AGPL3, BSD,MIT etc.
  summary:  #Overview
  description:Description
  license_family: BSD

instead of url for source You can also specify a relative path from the recipe directory with path.

The format of the virgin number follows PEP-440.

If you want to change the requirements and build number for each platform, Specify the selector and describe as follows.

build:
  number: 2 # [osx]
  number: 1 # [linux]

requirements:
  build:
    - ncurses
    - dbus # [osx]
    - jpeg # [linux]

The selector is given in the format `` `# [value] ```, and the value can be specified as follows.

Build script

What is a build script? Required to build a package A script with instructions.

#!/bin/bash

$PYTHON setup.py install

# Add more build steps here, if they are necessary.

# See
# http://docs.continuum.io/conda/build.html
# for a list of environment variables that are set during the build process.

According to the source availability information described in the metadata The conda build command downloads Extract it to your working directory and then call the build script.

The working directory is under $ HOME / conda / x64 / conda-build, Created with a * package name-epoch time * directory name.

When installing when creating a package Under the working directory, go to the directory starting with _b_env_ It will be installed in.

/home/iisaka/conda/x64/conda-bld/lftp_1485460657191
├── _b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl
│   ├── bin
│   ├── conda-meta
│   ├── include
│   ├── lib
│   ├── share
│   └── ssl
└── work
    └── lftp-4.7.5

Pre / post processing

If you want to process before and after the conda package is installed, Create the following script.

• pre-link.sh --Run before the package is installed
• post-link.sh --Run after the package is installed
• pre-unlink.sh --Run before the package is uninstalled

These scripts are executed after the following environment variables are set.

In pre / post processing, nothing is output to standard output or standard error output, It is stored in $ PREFIX / .messages.txt.

Environment variables in the Conda build process

The following environment variables are set internally in Conda, You can refer to it in the build script.

Environment variables in git

If you specify a git repository for source with git_url or path, the following environment variables are also set:

Environment variables in mercurial

If you specify a mercurial repository for source, The following environment variables are also set:

Inheritance of environment variables

On the Linux platform, in an environment that calls conda build other than the above, No variables are inherited. By adding it to meta.yaml You can add environment variables to inherit.

build:
  script_env:
   - TMPDIR
   - LD_LIBRARY_PATH # [linux]

If the inherited variable is missing from the build shell environment That variable remains unallocated, A warning is displayed stating that the value has not been set.

Note: Inheriting environment variables It can be difficult for others to reproduce the binaries from the source using recipes. Use this feature with caution or avoid it at all.

Environment variables that affect the build process

Environment variables in build features

Even if the features have the same name and version It is a mechanism that allows you to register the difference between packages. The packages specified by track_features will also be installed.

The feature is not the package itself These are the characteristics of the environment in which the package is installed. For example, if mkl is specified in track_features, Regular numpy removed and mkl featured numpy package installed This is because the conda automatically changes according to the specified feature, such as.

Build tips

When trying to build a package, multiple packages with dependencies You may need to build it.

Make sure the recipe directory name is the same as the package name specified in the metafile.

This way, when you build the package The same directory as the dependent package If it is in the current directory It will automatically build this package for you.

Let's make the PGI compiler a Conda package

Since 2016, the community version of the PGI compiler has been available for free, so Let's make this a Conda package.

numactl package

First, create the numactl package needed by the PGI compiler.

Create the same directory as the package name numactl and Place the metadata file and the bulld script there.

$ mkdir numactl 

** Metadata file **

package:
    name: numactl
    version: 2.0.11

source:
    fn:  numactl-2.0.11.tar.gz
    url: https://github.com/numactl/numactl/archive/v2.0.11.tar.gz
    md5: b56d2367217cde390b4d8087e00773b8

requirements:
    build:
      - m4
      - autoconf
      - automake
      - libtool
      - pkg-config
    run:

about:
    home: http://oss.sgi.com/projects/libnuma/
    summary: NUMA support for Linux
    license: GPL2.0

** Build script **

#!/bin/bash

export CFLAGS="${CFLAGS} -I${PREFIX}/include"
export CXXFLAGS="${CFLAGS}"
export CPPFLAGS="-I${PREFIX}/include"
export LDFLAGS="${LDFLAGS} -L${PREFIX}/lib"

bash autogen.sh

./configure --help
./configure \
    --prefix="${PREFIX}"   \
    --enable-static

make -j${CPU_COUNT}
make install

build numactl package

$ conda build numactl

PGI compiler

Since the notation of jinja2 can be used in the metafile, Generated by the value of the environment variable at build time as follows I try to switch packages.

Set default values for dependent packages

{% set require1 = "" %}
{% set require2 = "" %}
{% set numactl  = "" %}

#Holds the value of the environment variable HOME
{% set home = environ.get("HOME") %}

#Environment variable PGI_Holds the value of COMPONENT
{% set component = environ.get("PGI_COMPONENT") %}

# PGI_The package target is divided according to the value given by COMPONENT.
{% if component == "openmpi" %}
{% set name = "pgi-openmpi" %}
{% set require1 = "- pgi-compiler" %}
{% elif component == "cuda80" %}
{% set name = "pgi-cuda80" %}
{% set require1 = "- pgi-compiler" %}
{% set require2 = "- pgi-java" %}
{% elif component == "cuda75" %}
{% set name = "pgi-cuda75" %}
{% set require1 = "- pgi-compiler" %}
{% set require2 = "- pgi-java" %}
{% elif component == "examples" %}
{% set name = "pgi-examples" %}
{% set require1 = "- pgi-compiler" %}
{% elif component == "java" %}
{% set name = "pgi-java" %}
{% else %}
{% set name = "pgi-compiler" %}
#Request the numactl package only for the compiler
{% set numactl = "- numactl" %}
{% endif %}

package:
  name: {{ name }}
  version: 17.4

source:
#TAR ball downloaded from PGI site in advance$Save to HOME
  url: file://{{ home }}/pgilinux-2017-174-x86_64.tar.gz
patch:
  - fix-install.patch

build:
  number: 0
  binary_relocation: false

requirements:
  build:
    - perl
    {{ numactl }}
  run:
    {{ numactl }}
    {{ require1 }}
    {{ require2 }}

about:
  home: http://www.pgroup.com/products/community.htm
  license: PGI Eng-User License
  license_file: LICENSE.txt
  summary: "A no-cost license to a recent release of the PGI Fortran, C and C++ compilers and tools for multicore CPUs and NVIDIA Tesla GPUs, including all OpenACC, OpenMP and CUDA Fortran features."

The build script looks like this:

#!/bin/bash

#Set parameters for install script
export PGI_SILENT=true
export PGI_ACCEPT_EULA=accept
export PGI_INSTALL_DIR=${PREFIX}/pgi
export PGI_INSTALL_NVIDIA=false
export PGI_INSTALL_AMD=false
export PGI_INSTALL_JAVA=false
export PGI_INSTALL_MPI=false
export PGI_MPI_GPU_SUPPORT=false
export PGI_INSTALL_MANAGED=false

#Create an installation directory
[ -d ${PREFIX}/pgi ] || mkdir ${PREFIX}/pgi

# post-link.sh / pre-unlink.If sh is in the recipe directory, delete it
[ -f ${RECIPE_DIR}/post-link.sh ]  && rm -f ${RECIPE_DIR}/post-link.sh
[ -f ${RECIPE_DIR}/pre-unlink.sh ] && rm -f ${RECIPE_DIR}/pre-unlink.sh
Extract the file according to the package name
case "${PKG_NAME}" in
*-cuda75)
    tar -zxvf install_components/linux86-64.pgicuda.tar.gz \
              -C ${PREFIX}/pgi
    rm -rf ${PREFIX}/pgi/linux86-64/2017/cuda/8.0
    exit 0
;;
*-cuda80)
    tar -zxvf install_components/linux86-64.pgicuda8.0-libnvvp.tar.gz \
              -C ${PREFIX}/pgi
    tar -zxvf install_components/linux86-64.pgicuda.tar.gz \
              -C ${PREFIX}/pgi
    rm -rf ${PREFIX}/pgi/linux86-64/2017/cuda/7.5
    exit 0
;;
*-compiler)
    export PGI_INSTALL_MANAGED=true
    export PGI_INSTALL_AMD=true
    #Post only for compiler-link.sh / pre-unlink.enable sh
    [ -f ${RECIPE_DIR}/scripts/post-link.sh ]  && \
         cp ${RECIPE_DIR}/scripts/post-link.sh ${RECIPE_DIR}/post-link.sh
    [ -f ${RECIPE_DIR}/scripts/pre-unlink.sh ] && \
         cp ${RECIPE_DIR}/scripts/pre-unlink.sh ${RECIPE_DIR}/pre-unlink.sh
;;
*-openmpi)
    export PGI_INSTALL_MPI=true
    mkdir -vp workdir
    tar -zxvf install_components/openmpi-1.10.2_2017_x86_64.tar.gz \
              -C workdir
    tar -zxvf workdir/linux86-64.openmpi-1.10.2.tar.gz \
              -C ${PREFIX}/pgi/
    tar -zxvf install_components/scalapack-2.0.2_2017_x86_64.tar.gz \
              -C ${PREFIX}/pgi/

    mkdir -vp ${PREFIX}/pgi/modulefiles
    perl workdir/openmpi.pl \
         -libdir ${PREFIX}/pgi/linux86-64/2017/mpi/openmpi-1.10.2 \
         -openmpiver 1.10.2 \
         -install ${PREFIX}/pgi/modulefiles/ \
         -release 2017
    exit 0
;;
*-java)
    export PGI_INSTALL_JAVA=true
    mkdir -vp ${PREFIX}/pgi/linux86-64/17.4/java
    tar -zxvf install_components/common/jre-8u112-linux-x64.tar.gz \
       -C ${PREFIX}/pgi/linux86-64/17.4/java

    mkdir -vp ${PREFIX}/pgi/linux86-64/17.4/bin
    cp install_components/linux86-64/17.4/bin/*.jar \
       ${PREFIX}/pgi/linux86-64/17.4/bin
    cp install_components/linux86-64/17.4/bin/pgtjavarc \
       ${PREFIX}/pgi/linux86-64/17.4/bin
    exit 0
;;
*-example)
    tar -zxvf install_components/linux86-64.examples.tar.gz \
              -C ${PREFIX}/pgi
    exit 0
;;
esac

./install

#Delete unnecessary files
rm -rf ${PREFIX}/pgi/linux86-64/17.4/cray
rm -rf ${PREFIX}/pgi/linux86-64/2017/cray
rm -rf ${PREFIX}/pgi/linux86-64/2017/examples/

[ "${PGI_INSTALL_MPI}" != "true" ] && \
    rm -rf ${PREFIX}/pgi/linux86-64/17.4/lib/scalapack

[ "${PGI_INSTALL_MPI}" != "true" ] && {
    rm -rf ${PREFIX}/pgi/linux86-5/17.4/bin/*.jar
    rm -rf ${PREFIX}/pgi/linux86-5/17.4/bin/pgtjavarc
}

In the recipe directory scripts / post-link.sh Create scripts / pre-unlink.sh.

scripts/post-link.sh

#!/bin/bash

mkdir -p ${PREFIX}/etc/conda/deactivate.d
mkdir -p ${PREFIX}/etc/conda/activate.d

cat <<EOF > ${PREFIX}/etc/conda/deactivate.d/${PKG_NAME}.sh
export PATH=\${PATH_pgi}
export MANPATH=\${MANPATH_pgi}
export LD_LIBRARY_PATH=\${LD_LIBRARY_PATH_pgi}
unset PATH_pgi
unset MANPATH_pgi
unset LD_LIBRARY_PATH_pgi
EOF

cat <<EOF > ${PREFIX}/etc/conda/activate.d/${PKG_NAME}.sh
export PGI=${PREFIX}/pgi/linux86-64/${PKG_VERSION}
export CC=\${PGI}/bin/pgcc
export FC=\${PGI}/bin/pgfortran
export F90=\${PGI}/bin/pgf90
export F77=\${PGI}/bin/pgf77
export CPP="\${PGI}/bin/pgcc -E"
export CXX=\${PGI}/bin/pgc++
#
export PATH_pgi=\${PATH}
export PATH=\${PGI}/bin:\${PATH}
#
export MANPATH_pgi=\${MANPATH}
export MANPATH=\${PGI}/man:\${MANPATH}
#
export LD_LIBRARY_PATH_pgi=\${LD_LIBRARY_PATH}
export LD_LIBRARY_PATH=\${PGI}/lib:\${LD_LIBRARY_PATH}
EOF

scripts/pre-unlink.sh

#!/bin/sh

rm -f ${PREFIX}/etc/conda/deactivate.d/${PKG_NAME}.sh
rm -f ${PREFIX}/etc/conda/activate.d/${PKG_NAME}.sh

Patch file specified in the metadata file

fix-install.patch

diff -Nru pgilinux-2017-174-x86_64.orig/install_components/install pgilinux-2017-174-x86_64/install_components/install
--- pgilinux-2017-174-x86_64.orig/install_components/install	2017-04-22 02:16:31.000000000 +0900
+++ pgilinux-2017-174-x86_64/install_components/install	2017-06-01 14:30:18.902018374 +0900
@@ -739,27 +739,7 @@
             cd ..
             rm -rf scalapack
 
-            numa_lib=""
-            if test -e /usr/lib64/libnuma.so ; then
-                numa_lib=/usr/lib64/libnuma.so
-            elif test -e /usr/lib64/libnuma.so.1 ; then
-                numa_lib=/usr/lib64/libnuma.so.1
-            elif test -e /lib64/libnuma.so ; then
-                numa_lib=/lib64/libnuma.so
-            elif test -e /lib64/libnuma.so.1 ; then
-                numa_lib=/lib64/libnuma.so.1
-            elif test -e /usr/lib/x86_64-linux-gnu/libnuma.so ; then
-                numa_lib=/usr/lib/x86_64-linux-gnu/libnuma.so
-            elif test -e /usr/lib/x86_64-linux-gnu/libnuma.so.1 ; then
-                numa_lib=/usr/lib/x86_64-linux-gnu/libnuma.so.1
-            fi
-            if test "numa_lib" != "" ; then
-                ln -sf $numa_lib $INSTALL_DIR/linux86-64/$REL_VERSION/mpi/openmpi-${OMPI_VERSION}/lib/libnuma.so
-            else
-                echo "WARNING: libnuma.so was not found on your system!"
-                echo "Open MPI requires libnuma.so to function correctly."
-                echo "Please install libnuma.so from your Linux distribution onto this system."
-            fi
+            numa_lib="${PREFIX}/lib/libnuma.so"
 
             if test "$ans" != "y" -a "$ans" != "yes" ; then
                 mpi_wrapper="$INSTALL_DIR/linux86-64/$REL_VERSION/mpi/openmpi-${OMPI_VERSION}/bin/env.sh"

Reference material

• Anaconda Python
  • https://repo.continuum.io/
• Conda building package
  • https://conda.io/docs/building/build.html
• PGI Compiler Community Edition
  • http://www.pgroup.com/products/community.htm