[Solution] When I try to connect to CloudSQL with GAE, I get an ImportError only when using dev_appserver.

I'm addicted to using Google App Engine (Python) for my research. I'm developing a web service using Flask with GAE + CloudSQL, and when I try to connect to CloudSQL from Flask using SQLAlchemy, I get an error. Apparently I'm getting an error ʻImportError: No module name _mysql when trying to import MySQLdb. What's more, this error only occurs when trying to check the behavior locally with dev_appserver.py`.

The error log is as follows. (I get an error when sqlalchemy.create_engine () is executed when accessing / user)

$ dev_appserver.py app.yaml                                                                             
INFO     2017-07-27 05:57:06,227 sdk_update_checker.py:231] Checking for updates to the SDK.
INFO     2017-07-27 05:57:06,541 api_server.py:297] Starting API server at: http://localhost:52746
INFO     2017-07-27 05:57:06,545 dispatcher.py:209] Starting module "test" running at: http://localhost:8080
INFO     2017-07-27 05:57:06,548 admin_server.py:116] Starting admin server at: http://localhost:8000
WARNING  2017-07-27 05:57:06,548 devappserver2.py:187] No default module found. Ignoring.
INFO     2017-07-27 05:57:17,859 server.py:125] Server initialized for threading.
ERROR    2017-07-27 05:57:17,875 wsgi.py:263] 
Traceback (most recent call last):
  File "/Users/test/Desktop/study/gae/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/Users/test/Desktop/study/gae/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/Users/test/Desktop/study/gae/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "/Users/test/Desktop/test/test/main.py", line 30, in <module>
    engine = create_engine(url, echo=True)
  File "/Users/test/Desktop/test/test/lib/sqlalchemy/engine/__init__.py", line 387, in create_engine
    return strategy.create(*args, **kwargs)
  File "/Users/test/Desktop/test/test/lib/sqlalchemy/engine/strategies.py", line 80, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/Users/test/Desktop/test/test/lib/sqlalchemy/dialects/mysql/mysqldb.py", line 110, in dbapi
    return __import__('MySQLdb')
  File "/Users/test/Desktop/test/test/lib/MySQLdb/__init__.py", line 19, in <module>
    import _mysql
  File "/Users/test/Desktop/study/gae/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 1024, in load_module
    raise ImportError('No module named %s' % fullname)
ImportError: No module named _mysql
INFO     2017-07-27 05:57:17,882 module.py:809] test: "GET /user HTTP/1.1" 500 -

(* Replace the part you want to keep secret with test)

ʻApp.yaml and ʻappengine_config.py are as follows.

app.yaml


module: test
runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: /favicon\.ico
  static_files: favicon.ico
  upload: favicon\.ico

- url: .*
  script: main.app

libraries:
- name: MySQLdb
  version: "1.2.5"

appengine_config.py


from google.appengine.ext import vendor

# Add any libraries install in the "lib" folder.
vendor.add('lib')

The libraries are installing in the lib directory.

requirements.txt


Flask==0.10.1
Flask_SocketIO
itsdangerous==0.24
Jinja2==2.7.3
MarkupSafe==0.23
python-engineio
python-socketio
six==1.9.0
Werkzeug==0.10.4
sqlalchemy==1.1.9
mysql-python==1.2.5
$ sudo pip install -t lib -r requirements.txt

As a test, I imported MySQLdb in my local Python and got no error.

Python 2.7.9 (default, May  9 2017, 14:55:04) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>> 

As a trial, I tried raising it to the GAE environment as a dev version and it was deployed without problems, I was able to confirm the connection to CloudSQL without any problem.

$ appcfg.py -A test-project -V dev update app.yaml
03:15 PM Host: appengine.google.com
03:15 PM Application: test-project (was: None); module: test; version: dev (was: None)
03:15 PM Starting update of app: test-project, module: test, version: dev
03:15 PM Getting current resource limits.
03:15 PM Scanning files on local disk.
03:15 PM Scanned 500 files.
03:15 PM Cloning 544 application files.
03:15 PM Compilation starting.
03:16 PM Compilation completed.
03:16 PM Starting deployment.
03:16 PM Checking if deployment succeeded.
03:16 PM Deployment successful.
03:16 PM Checking if updated app version is serving.
03:16 PM Completed update of app: test-project, module: test, version: dev
$

_mysql is lost only when deployed to the local environment with dev_appserver.py.

We will continue to solve the problem. (It's a pain if you can't check the behavior of css or js locally)


(Addition: 2017/07/27 18:50)

_mysql is lost only when deployed to the local environment with dev_appserver.py.

When I went around this, I ran into the following stackoverflow.

google app engine - how to add lib folder?

appengine_config.py will run in the GAE environment or when you run dev_appserver, which emulates the GAE environment. However if you're ever running outside of a GAE environment, make sure your PYTHONPATH includes the lib folder you want to import from.

In other words, ʻappengine_config.py will not be executed when dev_appserver.pyis executed locally, so add thelib directory to PYTHONPATH`.

Actually set PYTHONPATH exactly as it is,

$ export PYTHONPATH=/Users/test/Desktop/test/test/lib:$PYTHONPATH

It worked when I ran it.

(After that, I read the official documentation, but I couldn't find anything that clearly states the relationship between dev_appserver.py and ʻappengine_config.py` ... orz)

I'm glad I solved it for the time being. Problems may be sorted out by summarizing them in sentences.

Recommended Posts

[Solution] When I try to connect to CloudSQL with GAE, I get an ImportError only when using dev_appserver.
I get an error when I try to raise Python to 3 series using pyenv on Catalina
I get an error when trying to install maec 4.0.1.0 with pip
When I get an error with PyInstaller
I get a UnicodeDecodeError when trying to connect to oracle with python sqlalchemy
I tried to get an AMI using AWS Lambda
I got an error when using Tensorboard with Pytorch
When I try to install mysqlclient with Django, I get error: command'gcc' failed with exit status 1.
When I try to push with heroku, it doesn't work
I get [Error 2055] when trying to connect to MySQL on Heroku
When I try to connect to MySQL with mysql-connector-python, I cannot connect with the error "SSL connection error: SSL_CTX_set_tmp_dh failed"
A memorandum when I tried to get it automatically with selenium
When I get an error with Pylint in Atom on Windows
When you want to send an object with requests using flask
I get an ImportError on appengine_config.py
Try to get statistics using e-Stat
I got an error when trying to install Xgboost and its solution
I want to create an Ubuntu chrome User Profile with Colab only
When I try to go back using chainer, it fits a little
Try to poke DB on IBM i with python + JDBC using JayDeBeApi
I get an error with import pandas.
I tried to get started with Hy
Try to generate an image with aliasing
When I try to connect django and postgresql with Docker, I get the error "django.db.utils.OperationalError: could not translate host name" db "to address: Name or service not known"
Workaround if you get an error when trying to install PySide with pip
Three things I was addicted to when using Python and MySQL with Docker
When I connect to a remote Jupyter Server with VScode, it's remote but local
When I tried to connect with SSH, I got a warning about free space.
What to do if you get an error when installing python with pyenv
When I try to use Jupiter notebook on Mac, I can only select python2
I tried to create an environment to check regularly using Selenium with AWS Fargate
I got an error when saving with OpenCV
I tried to get an image by scraping
I want to be an OREMO with setParam!
I tried to get CloudWatch data with Python
How to deal with SessionNotCreatedException when using Selenium
I get a UnicodeDecodeError when running with mod_wsgi
Precautions when using google-cloud library with GAE / py
I tried to detect an object with M2Det!
Try to create an HTTP server using Node.js
I get an error with all yum commands
[Python] I want to use only index when looping a list with a for statement
What to do if you get an OpenSSL error when installing Python 2 with pyenv
Every time I try to read a csv file using pandas, I get a numpy error.
When I try to import pandas on macOS I get the error No module named'_bz2'
I want to get an error message in Japanese with django Password Change Form
What to do if you get an Import Error when importing matplotlib with Jupyter
I want to extract only pods with the specified label using Label Selector in Client-go
When I try to update the data on DynamoDB with Python (boto3), I get "ExpressionAttributeNames contains invalid key: Syntax error; key: <key name>"