[Python] How to deal with module errors

Even though I worked on building an environment overnight, the module error (Import Error) was not resolved and I was about to become a dead man. The cause of the error was the baptism that the path was different, the library you wanted to use did not exist, or the library you were using was old, but when you are worried, such a problem really occurs in the first place. It was a trial and error process, even if it was unclear. What should I do to prevent such pain from being experienced again? Record the learning of environment construction here.

  1. Where is the execution script in the path?
  2. Where is the path of the library that the execution script is trying to reference?
  3. Is there a required library for that library?
  4. Are the versions of those libraries correct?

In this case, it is assumed that you create a Jenkins server with GCP, get a python script that operates BigQuery from Github, and execute it regularly.

1. Where is the execution script in the path?

By knowing where the script in error is running, you can directly see if the file itself works in that environment.

For example, let's say that a module error occurs when automatically acquiring and executing the source code placed on Github by using Jenkins installed on a server such as GCP (GCE) instead of running the script directly locally. .. jenkins_no5.png

Jenkins (console output)


Started by timer
Running as SYSTEM
Building in workspace /opt/bitnami/jenkins/jenkins_home/workspace/test_py
...
[test_py] $ /bin/bash /tmp/jenkins7138111450797694063.sh
Traceback (most recent call last):
  File "my_test_script.py", line 1, in <module>
    from google.cloud import bigquery
ModuleNotFoundError: No module named 'google.cloud'
Build step 'Execute shell' marked build as failure
Finished: FAILURE

Jenkins will automatically create a directory to put the retrieved scripts. You can identify the location by looking at the log, so you can move it directly to verify it.

CloudShell (moving directories and running scripts)


user:~$ cd /opt/bitnami/jenkins/jenkins_home/workspace/test_py
#Move to the directory created automatically by Jenkins according to the job name

user:/opt/bitnami/jenkins/jenkins_home/workspace/test_py$ python my_test_script.py
#Execution of script obtained from Github

2. Where is the path of the library that the execution script is trying to reference?

If the path where the actual library is placed and the path that the execution script loads to refer to the library do not match, the library cannot be loaded and an error occurs.

For example, there are many cases where "path does not pass" = the path where the actual library is placed is not set in the PATH environment setting variable (the path where the library is installed cannot be read by the script). .. In addition, it also occurs when a virtual environment is set up and commands are executed. Specifically, when executing a script after setting up a virtual environment such as venv, the library to be used is searched only from under that virtual environment. In other words, a library installed without setting up a virtual environment cannot be read by a script running under a virtual environment, and conversely, a library installed without setting up a virtual environment is used by a script running without setting up a virtual environment. Cannot result in a module error.

CloudShell (example of incorrect installation method causing module error)


$ cd my_directory
$ pip install --upgrade google-cloud-bigquery
#Install the library outside the virtual environment

$ python3 -m venv env
$ source env/bin/activate
(env) $ python my_test_script.py
#Enable and run virtual environment

Traceback (most recent call last):
  File "test.py", line 1, in <module>
    from google.cloud import bigquery
ImportError: No module named google.cloud
#Module error occurred

Therefore,

--Which path is the library installed in? ――Is that path readable by a script?

Need to be confirmed.

CloudShell (existence of library and installation path that can be seen from the log at the time of library installation)



$ pip install --upgrade google-cloud-bigquery
Requirement already satisfied: google-cloud-bigquery in /usr/local/lib/python3.7/dist-packages (4.56.0)
#It is said that it is already installed,
#In the first place "/usr/local/lib/python3.7/dist-"packages"
#Script execution environment(Virtual environment)It's not a pass, so it's NG.

$ source env/bin/activate
(env)$ pip install --upgrade google-cloud-bigquery
Requirement already satisfied: proto-plus>=1.10.0 in ./env/lib/python3.7/site-packages (from google-cloud-
bigquery) (1.13.0)
#Already a script execution environment(Under virtual environment)It is OK because it is installed in.

CloudShell (quick way to check the library)



(env) account:~$ pip list
Package                  Version  
------------------------ ---------
...
google-api-core          1.25.0   
google-auth              1.24.0   
google-cloud-bigquery    2.6.2    
google-cloud-core        1.5.0    
google-crc32c            1.1.1    
google-resumable-media   1.2.0    
googleapis-common-protos 1.52.0   
...
#Already "google-cloud-"bigquery" is installed, so it's OK

CloudShell (quick library confirmation method 2)


(env) account:~$ pip list
$ pip freeze
...
google-api-core==1.25.0
google-auth==1.24.0
google-cloud-bigquery==2.6.2
google-cloud-core==1.5.0
google-crc32c==1.1.1
google-resumable-media==1.2.0
googleapis-common-protos==1.52.0
...
#Already (ry

CloudShell (library path and existence that can be confirmed from PythonPATH)


$ pip --version
pip 20.3.3 from /usr/local/lib/python3.7/dist-packages/pip (python 3.7)

$ python
>>> import sys
>>> print(sys.path)
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/u
sr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', 
'/usr/lib/python2.7/dist-packages']
>>> exit()

#The actual library location and the path when the python file loads the library are different
#And even the assumed major version is different
#After that, avoid it by setting up a virtual environment


(env) account:~$ python
>>> import sys
>>> print (sys.path)
['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/home/info/env/lib/python3.7/site-packages']
>>> exit()

(env) account:~$ cd /home/info/env/lib/python3.7/site-packages
(env) account:~/env/lib/python3.7/site-packages$ ls
cachetools                                       google_cloud_bigquery-2.6.2-py3.9-nspkg.pth   pyasn1-0.4.8.dist-info
cachetools-4.2.0.dist-info                       google_cloud_core-1.5.0.dist-info             pyasn1_modules
certifi                                          google_cloud_core-1.5.0-py3.9-nspkg.pth       pyasn1_modules-0.2.8.dist-info
certifi-2020.12.5.dist-info                      google_crc32c                                 __pycache__
cffi                                             google_crc32c-1.1.1.dist-info                 pycparser
cffi-1.14.4.dist-info                            google_crc32c.libs                            pycparser-2.20.dist-info
_cffi_backend.cpython-37m-x86_64-linux-gnu.so    google_resumable_media-1.2.0.dist-info        pytz
cffi.libs                                        google_resumable_media-1.2.0-py3.9-nspkg.pth  pytz-2020.5.dist-info
chardet                                          grpc                                          requests
chardet-4.0.0.dist-info                          grpcio-1.34.1.dist-info                       requests-2.25.1.dist-info
_distutils_hack                                  idna                                          rsa
distutils-precedence.pth                         idna-2.10.dist-info                           rsa-4.7.dist-info
easy_install.py                                  pip                                           setuptools
google                                           pip-18.1.dist-info                            setuptools-51.3.3.dist-info
google_api_core-1.25.0.dist-info                 pkg_resources                                 six-1.15.0.dist-info
google_api_core-1.25.0-py3.9-nspkg.pth           pkg_resources-0.0.0.dist-info                 six.py
googleapis_common_protos-1.52.0.dist-info        proto                                         urllib3
googleapis_common_protos-1.52.0-py3.8-nspkg.pth  protobuf-3.14.0.dist-info                     urllib3-1.26.2.dist-info
google_auth-1.24.0.dist-info                     protobuf-3.14.0-py3.7-nspkg.pth               wheel
google_auth-1.24.0-py3.9-nspkg.pth               proto_plus-1.13.0.egg-info                    wheel-0.36.2.dist-info
google_cloud_bigquery-2.6.2.dist-info            pyasn1

#The destination path and the location of the library match

3. Is there a required library for that library? Is the version correct?

If the library A used by the python file is trying to use another library B or C, library A may not be installed properly if B or C does not exist or the version is old. So, if you use the command pip install --upgrade <library> to update all the related library versions, you may be able to install successfully.

CloudShell (Errors and actions that depend on other libraries)


(env) $ pip install --upgrade google-cloud-bigquery
...
 ----------------------------------------
  Failed building wheel for proto-plus
  Running setup.py clean for proto-plus
  Running setup.py bdist_wheel for grpcio ... error
  Complete output from command /home/info/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-kmhkwvwd/grpcio/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-26um0boj --python-tag cp37:
  Found cython-generated files...
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help
  
  error: invalid command 'bdist_wheel'
  
  ----------------------------------------
  Failed building wheel for grpcio
  Running setup.py clean for grpcio
Failed to build proto-plus grpcio
Installing collected packages: six, pyasn1, rsa, cachetools, pyasn1-modules, google-auth, protobuf, googleapis-common-protos, pytz, urllib3, chardet, idna, certifi, requests, grpcio, google-api-core, google-cloud-core, proto-plus, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery
  Running setup.py install for grpcio ... -

#Installation failed. It looks like we need something related to something called a "wheel".


(env) $ pip install --upgrade wheel
Collecting wheel
  Downloading https://files.pythonhosted.org/packages/65/63/39d04c74222770ed1589c0eaba06c05891801219272
420b40311cd60c880/wheel-0.36.2-py2.py3-none-any.whl
Installing collected packages: wheel
Successfully installed wheel-0.36.2
#I was able to install a new one

(env) $ pip install --upgrade grpcio
Collecting grpcio
  Using cached https://files.pythonhosted.org/packages/81/5e/168a7fa23a025beed6b7daa0981ace55e394a136db
3082faed7d6cba4556/grpcio-1.34.1.tar.gz
Requirement already satisfied, skipping upgrade: six>=1.5.2 in ./env/lib/python3.7/site-packages (from 
grpcio) (1.15.0)
Building wheels for collected packages: grpcio
  Running setup.py bdist_wheel for grpcio ... -
done
  Stored in directory: /home/info/.cache/pip/wheels/e2/60/7c/617a7c5af21a5d60c41e66ddb55f31235ec7d3f3d2
2d943f51
Successfully built grpcio
Installing collected packages: grpcio
Successfully installed grpcio-1.34.1
...
#I was able to install a new one

(env) $ pip install --upgrade google-cloud-bigquery
Successfully installed cffi-1.14.4 google-api-core-1.25.0 google-cloud-bigquery-2.6.2 google-cloud-core
-1.5.0 google-crc32c-1.1.1 google-resumable-media-1.2.0 proto-plus-1.13.0 pycparser-2.20
#I was able to install the desired library

end. In the future, I will carefully look at the output logs one by one and squeeze the points listed above.

If you have any advice, please let us know in this article or Twitter account m (__) m

Recommended Posts

[Python] How to deal with module errors
How to deal with DistributionNotFound errors
How to deal with enum compatibility errors
How to deal with run-time errors in subprocess.call
How to deal with errors when installing Python and pip with choco
[Python] How to deal with pandas read_html read error
Python: How to use async with
How to deal with imbalanced data
How to deal with imbalanced data
How to get started with Python
How to use FTP with Python
How to calculate date with python
[EC2] How to deal with errors that selenium cannot execute (No module named selenium)
For beginners, how to deal with common errors in keras
[Python] How to display random numbers (random module)
How to work with BigQuery in Python
How to do portmanteau test with python
How to display python Japanese with lolipop
How to enter Japanese with Python curses
How to add python module to anaconda environment
How to install python3 with docker centos
How to deal with python installation error in pyenv (BUILD FAILED)
How to deal with errors when installing whitenoise and deploying to Heroku
How to install python
How to upload with Heroku, Flask, Python, Git (4)
How to deal with memory leaks in matplotlib.pyplot
How to read a CSV file with Python 2/3
How to enjoy programming with Minecraft (Ruby, Python)
How to do multi-core parallel processing with python
Strategy on how to monetize with Python Java
[Python] How to draw multiple graphs with Matplotlib
[Python] How to read excel file with pandas
How to crop an image with Python + OpenCV
How to deal with module'tensorflow' has no attribute'〇〇'
How to deal with SessionNotCreatedException when using Selenium
How to add a Python module search path
How to specify attributes with Mock of python
How to measure execution time with Python Part 1
How to use tkinter with python in pyenv
[Python] How to handle Japanese characters with openCV
[Python] How to compare datetime with timezone added
How to measure execution time with Python Part 2
How to create random numbers with NumPy's random module
How to deal with OAuth2 error when using Google APIs from Python
How to deal with SSL error when connecting to S3 with boto of Python
How to deal with old Python versions in Cloud9 made by others
Connect to BigQuery with Python
[2020.8 latest] How to install Python
How to convert / restore a string with [] in python
How to add help to HDA (with Python script bonus)
How to install Python [Windows]
How Python module import works
python3: How to use bottle (2)
[Python] How to draw a line graph with Matplotlib
How to scrape image data from flickr with python
How to do hash calculation with salt in Python
[Introduction to Python] How to iterate with the range function?
Explain in detail how to make sounds with python
[Python] How to use list 1
Connect to Wikipedia with Python
How to update Python Tkinter to 8.6