[Python] How to deal with module errors
Even though I worked on building an environment overnight, the module error (Import Error) was not resolved and I was about to become a dead man. The cause of the error was the baptism that the path was different, the library you wanted to use did not exist, or the library you were using was old, but when you are worried, such a problem really occurs in the first place. It was a trial and error process, even if it was unclear. What should I do to prevent such pain from being experienced again? Record the learning of environment construction here.
- Where is the execution script in the path?
- Where is the path of the library that the execution script is trying to reference?
- Is there a required library for that library?
- Are the versions of those libraries correct?
In this case, it is assumed that you create a Jenkins server with GCP, get a python script that operates BigQuery from Github, and execute it regularly.
1. Where is the execution script in the path?
By knowing where the script in error is running, you can directly see if the file itself works in that environment.
For example, let's say that a module error occurs when automatically acquiring and executing the source code placed on Github by using Jenkins installed on a server such as GCP (GCE) instead of running the script directly locally. ..
Jenkins (console output)
Started by timer
Running as SYSTEM
Building in workspace /opt/bitnami/jenkins/jenkins_home/workspace/test_py
...
[test_py] $ /bin/bash /tmp/jenkins7138111450797694063.sh
Traceback (most recent call last):
File "my_test_script.py", line 1, in <module>
from google.cloud import bigquery
ModuleNotFoundError: No module named 'google.cloud'
Build step 'Execute shell' marked build as failure
Finished: FAILURE
Jenkins will automatically create a directory to put the retrieved scripts. You can identify the location by looking at the log, so you can move it directly to verify it.
CloudShell (moving directories and running scripts)
user:~$ cd /opt/bitnami/jenkins/jenkins_home/workspace/test_py
#Move to the directory created automatically by Jenkins according to the job name
user:/opt/bitnami/jenkins/jenkins_home/workspace/test_py$ python my_test_script.py
#Execution of script obtained from Github
2. Where is the path of the library that the execution script is trying to reference?
If the path where the actual library is placed and the path that the execution script loads to refer to the library do not match, the library cannot be loaded and an error occurs.
For example, there are many cases where "path does not pass" = the path where the actual library is placed is not set in the PATH environment setting variable (the path where the library is installed cannot be read by the script). .. In addition, it also occurs when a virtual environment is set up and commands are executed. Specifically, when executing a script after setting up a virtual environment such as venv, the library to be used is searched only from under that virtual environment. In other words, a library installed without setting up a virtual environment cannot be read by a script running under a virtual environment, and conversely, a library installed without setting up a virtual environment is used by a script running without setting up a virtual environment. Cannot result in a module error.
CloudShell (example of incorrect installation method causing module error)
$ cd my_directory
$ pip install --upgrade google-cloud-bigquery
#Install the library outside the virtual environment
$ python3 -m venv env
$ source env/bin/activate
(env) $ python my_test_script.py
#Enable and run virtual environment
Traceback (most recent call last):
File "test.py", line 1, in <module>
from google.cloud import bigquery
ImportError: No module named google.cloud
#Module error occurred
- By the way, even if you set up a virtual environment, if you execute it with
sudo pip install <library name>, the home directory will be/root instead of under the virtual environment, so the script to be executed under the virtual environment is A module error will occur.
Therefore,
--Which path is the library installed in? ββIs that path readable by a script?
Need to be confirmed.
CloudShell (existence of library and installation path that can be seen from the log at the time of library installation)
$ pip install --upgrade google-cloud-bigquery
Requirement already satisfied: google-cloud-bigquery in /usr/local/lib/python3.7/dist-packages (4.56.0)
#It is said that it is already installed,
#In the first place "/usr/local/lib/python3.7/dist-"packages"
#Script execution environment(Virtual environment)It's not a pass, so it's NG.
$ source env/bin/activate
(env)$ pip install --upgrade google-cloud-bigquery
Requirement already satisfied: proto-plus>=1.10.0 in ./env/lib/python3.7/site-packages (from google-cloud-
bigquery) (1.13.0)
#Already a script execution environment(Under virtual environment)It is OK because it is installed in.
CloudShell (quick way to check the library)
(env) account:~$ pip list
Package Version
------------------------ ---------
...
google-api-core 1.25.0
google-auth 1.24.0
google-cloud-bigquery 2.6.2
google-cloud-core 1.5.0
google-crc32c 1.1.1
google-resumable-media 1.2.0
googleapis-common-protos 1.52.0
...
#Already "google-cloud-"bigquery" is installed, so it's OK
CloudShell (quick library confirmation method 2)
(env) account:~$ pip list
$ pip freeze
...
google-api-core==1.25.0
google-auth==1.24.0
google-cloud-bigquery==2.6.2
google-cloud-core==1.5.0
google-crc32c==1.1.1
google-resumable-media==1.2.0
googleapis-common-protos==1.52.0
...
#Already (ry
CloudShell (library path and existence that can be confirmed from PythonPATH)
$ pip --version
pip 20.3.3 from /usr/local/lib/python3.7/dist-packages/pip (python 3.7)
$ python
>>> import sys
>>> print(sys.path)
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/u
sr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages']
>>> exit()
#The actual library location and the path when the python file loads the library are different
#And even the assumed major version is different
#After that, avoid it by setting up a virtual environment
(env) account:~$ python
>>> import sys
>>> print (sys.path)
['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/home/info/env/lib/python3.7/site-packages']
>>> exit()
(env) account:~$ cd /home/info/env/lib/python3.7/site-packages
(env) account:~/env/lib/python3.7/site-packages$ ls
cachetools google_cloud_bigquery-2.6.2-py3.9-nspkg.pth pyasn1-0.4.8.dist-info
cachetools-4.2.0.dist-info google_cloud_core-1.5.0.dist-info pyasn1_modules
certifi google_cloud_core-1.5.0-py3.9-nspkg.pth pyasn1_modules-0.2.8.dist-info
certifi-2020.12.5.dist-info google_crc32c __pycache__
cffi google_crc32c-1.1.1.dist-info pycparser
cffi-1.14.4.dist-info google_crc32c.libs pycparser-2.20.dist-info
_cffi_backend.cpython-37m-x86_64-linux-gnu.so google_resumable_media-1.2.0.dist-info pytz
cffi.libs google_resumable_media-1.2.0-py3.9-nspkg.pth pytz-2020.5.dist-info
chardet grpc requests
chardet-4.0.0.dist-info grpcio-1.34.1.dist-info requests-2.25.1.dist-info
_distutils_hack idna rsa
distutils-precedence.pth idna-2.10.dist-info rsa-4.7.dist-info
easy_install.py pip setuptools
google pip-18.1.dist-info setuptools-51.3.3.dist-info
google_api_core-1.25.0.dist-info pkg_resources six-1.15.0.dist-info
google_api_core-1.25.0-py3.9-nspkg.pth pkg_resources-0.0.0.dist-info six.py
googleapis_common_protos-1.52.0.dist-info proto urllib3
googleapis_common_protos-1.52.0-py3.8-nspkg.pth protobuf-3.14.0.dist-info urllib3-1.26.2.dist-info
google_auth-1.24.0.dist-info protobuf-3.14.0-py3.7-nspkg.pth wheel
google_auth-1.24.0-py3.9-nspkg.pth proto_plus-1.13.0.egg-info wheel-0.36.2.dist-info
google_cloud_bigquery-2.6.2.dist-info pyasn1
#The destination path and the location of the library match
3. Is there a required library for that library? Is the version correct?
If the library A used by the python file is trying to use another library B or C, library A may not be installed properly if B or C does not exist or the version is old. So, if you use the command pip install --upgrade <library> to update all the related library versions, you may be able to install successfully.
CloudShell (Errors and actions that depend on other libraries)
(env) $ pip install --upgrade google-cloud-bigquery
...
----------------------------------------
Failed building wheel for proto-plus
Running setup.py clean for proto-plus
Running setup.py bdist_wheel for grpcio ... error
Complete output from command /home/info/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-kmhkwvwd/grpcio/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-26um0boj --python-tag cp37:
Found cython-generated files...
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: -c --help [cmd1 cmd2 ...]
or: -c --help-commands
or: -c cmd --help
error: invalid command 'bdist_wheel'
----------------------------------------
Failed building wheel for grpcio
Running setup.py clean for grpcio
Failed to build proto-plus grpcio
Installing collected packages: six, pyasn1, rsa, cachetools, pyasn1-modules, google-auth, protobuf, googleapis-common-protos, pytz, urllib3, chardet, idna, certifi, requests, grpcio, google-api-core, google-cloud-core, proto-plus, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery
Running setup.py install for grpcio ... -
#Installation failed. It looks like we need something related to something called a "wheel".
(env) $ pip install --upgrade wheel
Collecting wheel
Downloading https://files.pythonhosted.org/packages/65/63/39d04c74222770ed1589c0eaba06c05891801219272
420b40311cd60c880/wheel-0.36.2-py2.py3-none-any.whl
Installing collected packages: wheel
Successfully installed wheel-0.36.2
#I was able to install a new one
(env) $ pip install --upgrade grpcio
Collecting grpcio
Using cached https://files.pythonhosted.org/packages/81/5e/168a7fa23a025beed6b7daa0981ace55e394a136db
3082faed7d6cba4556/grpcio-1.34.1.tar.gz
Requirement already satisfied, skipping upgrade: six>=1.5.2 in ./env/lib/python3.7/site-packages (from
grpcio) (1.15.0)
Building wheels for collected packages: grpcio
Running setup.py bdist_wheel for grpcio ... -
done
Stored in directory: /home/info/.cache/pip/wheels/e2/60/7c/617a7c5af21a5d60c41e66ddb55f31235ec7d3f3d2
2d943f51
Successfully built grpcio
Installing collected packages: grpcio
Successfully installed grpcio-1.34.1
...
#I was able to install a new one
(env) $ pip install --upgrade google-cloud-bigquery
Successfully installed cffi-1.14.4 google-api-core-1.25.0 google-cloud-bigquery-2.6.2 google-cloud-core
-1.5.0 google-crc32c-1.1.1 google-resumable-media-1.2.0 proto-plus-1.13.0 pycparser-2.20
#I was able to install the desired library
end. In the future, I will carefully look at the output logs one by one and squeeze the points listed above.
If you have any advice, please let us know in this article or Twitter account m (__) m
Recommended Posts