Summary of python environment settings for myself [mac] [ubuntu]

There were many useless descriptions, so I rewrote it. This one finishes faster. http://qiita.com/miyamotok0105/items/5f26e4ae41f0e35ded16

Build an environment for deep learning study. I had a lot of trouble, so I put it together for myself. It doesn't seem to work very well, but I've included both anaconda and pyenv. Switch the environment with conda.

Preferences tool

-Pyenv, anyenv ... python version switching ・ Virtualenv ... Python library independent environment construction ・ Anaconda ... pyenv and virturalenv are combined ・ Brew ... mac library management tool ・ Pip ... python library management tool ・ Conda ... python library management tool

Common errors in environment construction

Not installed ➡︎ Install Different python version ➡︎ Switch No path specified ➡︎ Specify Wrong path ➡︎ Correct path Dependency problem ➡︎ Uninstall and reinstall

Rough installation flow

--Preferences on mac Install pyenv with brew Install by specifying anaconda, miniconda or python version with pyenv Build a python2 or 3 virtual environment with conda Install the library with pip, brew --Preferences on ubuntu Drop pyenv from git on the pyenv official page. Install by specifying anaconda, miniconda or python version with pyenv Build a python2 or 3 virtual environment with conda Install the library with pip --Preferences for windows Install 2 or 3 on the anaconda official page Install the library with pip https://www.continuum.io/downloads

1. Installation

Install brew

Mac program environment construction tool opencv etc. There are tools to install from pip and tools to install from brew. If you install from brew, you may not be able to call from python because the path does not pass unless you paste the link.

bash【mac】

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

setup tool Mac If it is not included in the default python, enter it. Necessary when pip is not included.

bash【mac】

curl https://bootstrap.pypa.io/ez_setup.py -o - | python

Upgrade setup tools

bash【mac】

pip install --upgrade setuptools

Install pyenv

Tool to switch python version

bash【mac】

brew install pyenv 

bash【ubuntu】

git clone https://github.com/yyuu/pyenv.git ~/.pyenv
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc

Confirm that the pyenv command responds

Official page https://github.com/yyuu/pyenv

Show the list that can be installed with pyenv

bash【mac】【ubuntu】

pyenv install -l

Install python from pyenv

bash【mac】

pyenv install 2.7.12

Specify the path after installation

In case of ~~ bash, add the writing to ~ / .bashrc.

I took the method of using only ~~ anaconda3 and not switching the version with peen from there. Create and switch the environment such as python2 system with the new version with conda create. ~~

bash [mac] [Example of using now]

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
export PATH="$PYENV_ROOT/versions/anaconda3-4.1.1/bin/:$PATH"    

bash【mac】

#In the case of local, to the executed forta.python-You can create a version file
pyenv local 2.7.12  #Switch the version under the current folder
pyenv global 2.7.12  #Switch versions in the entire folder
pyenv rehash #Reflect

Install the library with pip

pip is a python library management tool In ruby, gem. Nuget in C #.

bash【mac】

pip install hogehoge

Install libraries together

bash【mac】

pip install -r requirements.txt

Delete all pip

bash【mac】

pip freeze > piplist.txt
sudo pip uninstall -r piplist.txt

Version specification

bash【mac】

pip install six==“1.8.0”

Update pip itself

bash【mac】

pip install -U pip
         or
pip install --upgrade pip

pip bulk update

bash【mac】

pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs pip install -U
pip --version

Specify the installation destination of pip

bash【mac】

pip install --install-option="--prefix=$HOME/local" hoge

Uninstall

pip uninstall hogehoge

http://hacknote.jp/archives/19074/

ipython installation

bash【mac】

pip install ipython

running ipython

bash【mac】

ipython notebook

jupyter notebook If you want to use python based on GUI

bash【mac】

pip install -U jupyter

Run jupyter

bash【mac】

jupyter notebook

Installation of anaconda

anaconda is a set of libraries that are likely to be used. It has the same function as virtualenv and does not destroy the original environment.

Install a specific version of anaconda from pyenv

bash【mac】

pyenv install anaconda-hogehoge

Anaconda environment replication (2 system python environment)

bash【mac】

conda create -n py27 python=2.7 anaconda

Duplicate the same environment

bash【mac】

Clone an environment

Get out of the current anaconda environment

bash【mac】

source deactivate

Anaconda environment deleted

bash【mac】

conda remove --name py27 --all

Reinstall conda

bash【ubuntu】

rm -rf ~/.conda

Use the anaconda environment from a file

bash【mac】

conda env create -f environment.yml
conda list

Install another version

bash【mac】

conda create --name snakes python=3
source activate snakes
conda info --envs
python —-version

Python update in conda

bash【mac】

conda update python

Upgrade python within conda

bash【mac】

conda install python=3.5

Package installation

bash【mac】

conda install --name bunnies beautiful-soup

Update of conda itself

bash【mac】

conda update conda
conda update anaconda

All batch update

bash【mac】

conda update --all

install anaconda pip

bash【mac】

~/.pyenv/versions/anaconda3-2.4.1/bin/pip install [package name]

Updated anaconda pip

bash【mac】

~/.pyenv/versions/anaconda3-2.4.1/bin/pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs ~/.pyenv/versions/anaconda3-2.4.1/bin/pip install -U

Uninstall

conda uninstall hogehoge

git

.gitignore

!.gitkeep
*.DS_Store

install opencv

bash【mac】【ubuntu】version2

I haven't tried it but recently-n [Environment name]I think that was important.
conda install -c menpo opencv=2.4.11

conda install -c menpo opencv3=3.1.0 -n [Environment name]

bash【mac】【ubuntu】version3

I haven't tried it but recently-n [Environment name]I think that was important.
conda install -c menpo opencv3=3.1.0 -n [Environment name]

conda install -c https://conda.binstar.org/jjhelmus opencv Official page https://anaconda.org/menpo/opencv https://anaconda.org/menpo/opencv3

Check version

import cv2 
cv2.__version__

Uninstall

If there are many, erase them all
conda uninstall -c menpo opencv=2.4.11
conda uninstall opencv
conda uninstall opencv3
Check if everything has disappeared
conda list | grep opencv

Put a link

bash【mac】

brew link opencv3 --force

If you want to install contrib

• If you use brew, pass the path http://blog.amedama.jp/entry/2015/10/01/223056

bash【mac】

brew install opencv3 --with-contrib   #For python2 series
or
brew install opencv3 --with-contrib --with-python3  #For python3 series

tap is used to put something that is not an official library. Since it is switched by pyenv, cv is not read unless the path of the library is also switched by source.

bash【mac】

brew tap homebrew/science
brew install opencv
source ~/.bashrc 

bash【ubuntu】

git clone https://github.com/Itseez/opencv.git
cd opencv
mkdir build
cd build
cmake ..
make -j4 or make -j8
sudo make install

• I was really into it. When I put it in from make, opencv got an error everywhere, and once I put it back in from pyenv, it worked. I was addicted to it for about 5 hours. .. .. If it doesn't work, let's enter from conda.

~~git clone https://github.com/jayrambhia/Install-OpenCV.git cd Install-OpenCV/Ubuntu ./opencv_latest.sh~~

Erase all opencv2.4

bash【ubuntu】

sudo apt-get autoremove opencv-doc opencv-data libopencv-dev libopencv2.4-java libopencv2.4-jni python-opencv libopencv-core2.4 libopencv-gpu2.4 libopencv-ts2.4 libopencv-photo2.4 libopencv-contrib2.4 libopencv-imgproc2.4 libopencv-superres2.4 libopencv-stitching2.4 libopencv-ocl2.4 libopencv-legacy2.4 libopencv-ml2.4 libopencv-video2.4 libopencv-videostab2.4 libopencv-objdetect2.4 libopencv-calib3d2.4 

Make uninstall normally

bash【ubuntu】

make uninstall

In case of opencv build error

libdc1394 error: Failed to initialize libdc1394

Fixed the contents of CMakeLists. -D Set WITH_1394 = OFF. https://github.com/opencv/opencv/blob/master/CMakeLists.txt

reference http://worktoolsmith.com/ubuntu-opencv/

dlib installation

bash【mac】

sudo pip install lib

If you get an error, you can enter it with conda http://paper.hatenadiary.jp/entry/2016/09/04/173818

brew install cmake
brew install boost-python
conda install -c menpo dlib=18.18

Install tensorflow

Package dependent error. An error will occur if setuptools is already included.

Cannot remove entries from nonexistent file /Users/user/.pyenv/versions/anaconda3-4.1.1/lib/python3.5/site-packages/easy-install.pth

In this case, add --upgrade

bash【mac】

~/.pyenv/versions/anaconda3-4.1.1/bin/pip install --upgrade --ignore-installed $TF_BINARY_URL

An error may occur due to numpy dependencies

ImportError: numpy.core.multiarray failed to import http://qiita.com/LyricalMaestro0/items/fe5baa162978cb3f7030

Install chainer

Install by specifying the version sudo ~/.pyenv/versions/anaconda3-4.1.1/envs/py27/bin/pip install chainer==1.5.1

2. Version switching and confirmation command

Check the path of the python library

Check the installation path

bash【mac】

python -c "from distutils.sysconfig import get_python_lib; print (get_python_lib())"

or

bash【mac】

python
>>> import sys
>>> sys.path

Show list of updatable packages

bash【mac】

pip list -o
pip show <package>

Check version

bash【mac】

python --version
python -V

Switch and check versions with pyenv

Check the currently selected python version

bash【mac】

pyenv versions

Switch environment with pyenv

• It is better to switch the environment with conda without using it much here.

bash【mac】

pyenv local anaconda3.4.1

Magic after switching with pyenv

bash【mac】

pyenv rehash

Check the list of installed libraries

bash【mac】

pyenv install -l

Version switching and confirmation with anaconda

Library list

bash【mac】

conda list

Search library

bash【mac】

conda search <foo>

Environment list

bash【mac】

conda info -e
conda info --envs
conda env list

Environment switching

bash【mac】【ubuntu】

source activate hogehoge

• When colliding with pyenv

bash【mac】

source versions/anaconda3-2.5.0/bin/activate hogehoge

windows does not have a source command so activate directly

bash【windows】

activate hogehoge

Exit the environment

source deactivate

If you want to delete the whole anaconda, delete the anaconda folder as it is

Check python version

bash【mac】

conda search python

Package List

bash【mac】

conda list
conda list -n snowflakes

Error when switching

• When switching versions with pyenv, it may be fixed by returning conda to root and then switching. Mystery

Your PYTHONPATH points to a site-packages dir for Python 2.x bet you are running Python 3.x! Resolve python path by case export PYTHONPATH="/usr/local/lib/python2.7/site-packages/:PYTHONPATH" export PYTHONPATH="/usr/local/lib/python3.5/site-packages/:PYTHONPATH"

Learning

If you want to learn on the server side, using nohup is convenient because it runs in the background nohup ignores the hangup signal (SIGHUP, HUP), which is a forced termination command when the terminal is disconnected. Write like this.

nohup sh run_minist.sh &

nohup example

nohup ping 8.8.8.8 &
jobs
fg %1

python basics

Import files in higher level directories import sys,os sys.path.append(os.pardir) from a import *

Measures against garbled characters

For python3

echo $LANG It may be c when you do. export LANG=en_US.UTF-8 If you change to, Japanese will be displayed.

Settings that should not be done below

~~ [Caution] When pyenv and anaconda coexist, the path was switched by looking at .pyathon-version. I tried to rewrite only the version I was using, but stopped this method. (In the end, I got an error that the path did not pass, so I switched the environment only with anaconda and did not use pyenv switching.) ~~

bash [mac] [bad example]

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

if [ ! -e "./.python-version" ];
then
    echo not exits
else
    A=`cat ./.python-version`
    if [ "anaconda-2.4.0" = ${A} ];
    then
	echo switch anaconda-2.4.0
	export PYTHONPATH="/usr/local/lib/python2.7/site-packages/:$PYTHONPATH"
	export PATH="$PYENV_ROOT/versions/anaconda-2.4.0/bin/:$PATH"
    else
	echo switch anaconda3-4.1.1
	export PYTHONPATH="/usr/local/lib/python3.5/site-packages/:$PYTHONPATH"
	export PATH="$PYENV_ROOT/versions/anaconda3-4.1.1/bin/:$PATH"    
    fi
fi

Automatic rehash

~~ If it is troublesome to rehash every time you switch with pyenv ~~

bash【mac】

brew install homebrew/boneyard/pyenv-pip-rehash

If it never disappears

~~ It might be a little dangerous. At your own risk. After all, I put it back in from pyenv. You shouldn't do this either. sudo find / -name "opencv" -exec rm -if {} ;~~

windows edition

The ones that can be entered from the command line and the ones that can be entered from the installer are different and honestly painful. The python environment was installed directly with the anaconda installer. It is also being installed on ubuntu on windows.

gfortran Since matplotlib depended on fortran, I installed it with the installer. https://qiita.com/HirofumiYashima/items/846cd8e51408ab8c88f9

SciPy

I can't enter it with pip at all, so I dropped it from git in the form of whl and entered it directly. scipy‑0.19.0‑cp36‑cp36m‑win_amd64.whl python -m pip install D:\PythonInstaller\scipy-0.19.0-cp36-cp36m-win_amd64.whl

amd, x86 or 64 Is it win, linux or mac? Is it python2.7 or python3.5? I was careful about the details and downloaded it.

http://mstn.hateblo.jp/entry/2017/04/13/014519

dlib There was also conda. I ran to conda because boost didn't build at all.

conda install -c conda-forge dlib=19.4

https://github.com/davisking/dlib

About conda (windows)

An implementation that has a cache that is typical of tools like this, such as xcode's cocoapods. So rarely a library that also affects conda If you overwrite the library with the error, you need to uninstall conda itself or clear the cache. I can't even move with pip.

Cache location. If you erase all of this, you can take it again.

C:\Users\[username]\AppData\Local\Continuum\anaconda3\pkgs

By the way, the environment created by conda is as follows

C:\Users\[username]\AppData\Local\Continuum\anaconda3\envs