Precautions when using google-cloud library with GAE / py

Even if you are developing in a GAE / py environment, you can access the datastore of other projects, etc. To use Google's RPC API I think there are cases where you want to use the google-cloud-python library.

At that time, there were some points to note regarding the path, so make a note of them.

google-cloud-python

This library is for hitting the GCP API from GCE or locally, Because you are not using C? , GAE / py also works.

It used to be called gcloud-python, but around September 2016, namespace was renamed to gcloud-> google.cloud. gcloud-python (I'm writing this article because of this rename)

How to use

You can see the basic usage by looking at read the docs, Find google-cloud or the module you want to use from pypi and install it with pip.

$ pip install google-cloud-datastore

After that, the module will be expanded under google.cloud, so you can use it as it is.

from google.cloud import datastore

client = datastore.Client()
product_key = client.key('Product', 123)
print(client.get(product_key))

Conflict with appengine module

As you may have noticed if you read this far, In GAE / py, import the appengine module from under google.appengine and use it. Therefore, if you install google-cloud, Modules under google in the local virtualenv environment, It will be overwritten by google-cloud from GAE SDK.

At this time, if you execute python on virtualenv, Whether it's a REPL or a file execution such as a unit test,

>>> import google.appengine
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named appengine

I get an error like this:

Workaround

Conclusion

In the virtualenv lib / python2.7 / site-packages / _virtualenv_path_extensions.pth I added the module addition logic while adding the library path of google_appengine. Of course, it would be the same if you add it to your own .pth file.

_virtualenv_path_extensions.pth


# ...Below is the location of the python sdk/usr/local/google_appengine case
/usr/local/google_appengine
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join('/usr/local/google_appengine', *('google',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('google', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('google', [os.path.dirname(p)])));m = m or not has_mfs and sys.modules.setdefault('google', types.ModuleType('google'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
# ...

Contents

Since it is difficult to read in one line, insert a line break and look at the contents. Since I brought the description of other libraries of google, there is also a description of 3.5 correspondence, but GAE / py is 2.7 series, so I think it can be scraped.

import sys, types, os
has_mfs = sys.version_info > (3, 5) 
# google_Create google module path in app engine
p = os.path.join('/usr/local/google_appengine', *('google',))

#Extract Module object
# 3.5 cases
importlib = has_mfs and __import__('importlib.util')
has_mfs and __import__('importlib.machinery')
m = has_mfs and sys.modules.setdefault('google', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('google', [os.path.dirname(p)])))
# 2.7 cases
m = m or not has_mfs and sys.modules.setdefault('google', types.ModuleType('google'))
mp = (m or []) and m.__dict__.setdefault('__path__',[])
#Add path to module
(p not in mp) and mp.append(p)

Survey method

It's a survey or a reference. ..

Adding multiple modules to the same namespace without python I don't think it's very welcome. So I thought it would be difficult to solve it.

However, if you take a closer look at the google-cloud library, Packages for multiple pypi are installed. And they are added under the google and google.cloud modules. I thought I could fix the path in the same way, so I searched for the .pth file and found the answer.

I don't really understand the style of .pth files,

By the way

In production

It seems that the above measures cannot be added to GAE / py, When I deployed, both modules were properly added to the google module. As expected. (I want you to distribute app engine sdk with pypi like google-cloud library)

nose Also, there was an error in python -m unittest discover etc. When I run nosetests, the test completes without any problems.

This is because the nose library wraps the unittest and before running the test It seemed to be because I was modifying PYTHONPATH.

-Prevent tampering with nose sys.path

In CircleCI, test runner defaults to nose I was late to realize that the path conflict wasn't resolved because the test passed successfully. I think that there is no problem if you are doing nosetests locally, It is a problem when debugging with REPL etc., so I think it is better to solve it.

reference

I didn't understand much about PYTHONPATH and .pth files, so I searched a lot. .. -What is PYTHONPATH --How to add default path in Python --Control the Python library path -[210: Control Python library loader](http://lightson.dip.jp/zope/ZWiki/210Python_e3_81_ae_e3_83_a9_e3_82_a4_e3_83_96_e3_83_a9_e3_83_aa_e3_83_ad_e3_83_bc_e3_e3_8_e3_e3_e3_e3_e3_e3_e3_e3_e3_e3_ -Search for Python module search path

Recommended Posts

Precautions when using google-cloud library with GAE / py
Precautions when using six with Python 2.5
Precautions when using Chainer
Precautions when using sqlite3 on macOS Sierra (10.12) with multiprocessing
When using optparse with iPython
Precautions when using TextBlob trait analysis
Precautions when installing tensorflow with anaconda
Precautions when using the urllib.parse.quote function
Precautions when using phantomjs from python
When using MeCab with virtualenv python
Precautions when solving DP problems with Python
Precautions when using for statements in pandas
[Personal memo] julia --Using Python library with julia using PyCall
Precautions when using OpenCV from Power Automate Desktop
About WARNING when packaging with pyinstaller using pyocr
Precautions when dealing with control structures in Python 2.6
Secret key generation when using EncryptedCookieStorage with aiohttp_session
How to deal with SessionNotCreatedException when using Selenium
Precautions when using tf.keras.layers.TimeDistributed for tf.keras custom layer
[Web development with Python] Precautions when saving cookies
What are you using when testing with Python?
GAE / py deployment
Call your own C library with Go using cgo
Precautions when installing a hierarchical include directory with waf
Precautions when dealing with ROS MultiArray types in Python
Troublesome story when using Python3 with VScode on ubuntu
Use of Google Cloud Storage (GCS) with "GAE / Py"
I got an error when using Tensorboard with Pytorch
When using Union type with pydantic, combine it with Literal
Precautions when operating with string for TmeStampType of PySpark
[Solution] When I try to connect to CloudSQL with GAE, I get an ImportError only when using dev_appserver.