[DOCKER] A story about cross-compiling a python package for AWS Lambda and deploying it serverless

I would like to know if there is a smarter way.

Trigger

I imported mysqlclient to connect to RDS (mysql / aurora) from AWS Lambda. It worked locally, so deploy it with serverless. When I test run it from the AWS Management Console, it doesn't work.

Apparently, mysqlclient contains native code, so I'm getting an error trying to run an image built on a Mac on linux.

It is a story that I tried various things to solve this area.

First try deploying as it is

By the way, the external package of python is managed by serverless-python-requirements.

--Reference: Management of external modules using Serverless Framework plugin

My service like this

ore-service
├── handler.py
├── requirements.py
├── requirements.txt
└── serverless.yml

Suppose that the content is like this.

requirements.txt


mysqlclient

handler.py


#!/use/bin/env python
# -*- coding: utf-8 -*-

import requirements
import MySQLdb

def lambda_handler(event, context):
    con = MySQLdb.connect(host='〜', db='〜', user='〜', passwd='〜', charset='utf8')
    cur = con.cursor()
    cur.execute("SELECT *FROM Somehow ~")

    ....(Abbreviation)

I get an error when I try to test run AWS Lambda from the AWS console with sls deploy. Log output has the following output.

Unable to import module 'handler': /var/task/_mysql.so: invalid ELF header

Oh, I see. I'm trying to run _mysql.so built on a Mac on linux.

Try using dockerizePip in serverless-python-requirements

serverless-python-requirements has an option called dockerizePip, and if you set it to true, it will cross-compile using the docker-lambda image. I will try it immediately.

serverless.yml looks like this:

serverless.yml


...(Abbreviation)

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: true

...(Abbreviation)

So, unfortunately, I got an error when deploying.

$ sls deploy
Serverless: Installing required Python packages...

  Error --------------------------------------------------

     Command "python setup.py egg_info" failed with error
     code 1 in /tmp/pip-build-4MzA_g/mysqlclient/

This seems to be because the python-devel and mysql-devel needed to build mysqlclient are not in the docker image.

Cross-compile with lambci / lambda

[Source] of serverless-python-requirements (https://github.com/UnitedIncome/serverless-python-requirements/blob/master/index.js) When I looked at it, I could understand what I was doing. It's just like starting a container and doing pip install.

Then you can start the container yourself, enter with bash and install the necessary libraries. It looks like the following.

$ docker run -it --rm -v "$PWD":/var/task "lambci/lambda:build-python2.7" bash

bash-4.2# cd /var/task
bash-4.2# yum -y install python-devel mysql-devel
bash-4.2# pip install mysqlclient -t .
bash-4.2# cp /usr/lib64/mysql/libmysqlclient.so.18 .

Mount / var / task of the container to the local current. This area is a plagiarism of the processing around installRequirements () of the source that I saw earlier.

Then install the required libraries and pip. I also need libmysqlclient.so.18, so copy it.

serverless uploads the files and folders in the location of serverless.yml as a zip, so you can sls deploy as it is.

(Delete the dockerizePip: true that you added to serverless.yml earlier)

This will load the mysqlclient built for linux from within lambda and you can safely access your RDS.

Use different packages to load according to the execution environment

A problem occurred here. I can't run locally because I have a linux image of mysqlclient locally.

$ python handler.py
Traceback (most recent call last):
  File "handler.py", line 18, in <module>
    import MySQLdb
  File "./MySQLdb/__init__.py", line 19, in <module>
    import _mysql
ImportError: dlopen(./_mysql.so, 2): no suitable image found.  Did find:
        ./_mysql.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00

Therefore, it is possible to switch the loading mysqlclient binary. The strategy is to put linux ELF under ./third-party/ and load it first if the execution environment is other than mac. The project root will not get dirty with external package folders.

The directory structure is as follows.

ore-service
├── third-party
│   ├── MySQLdb
│   ├── _mysql.so
│   ├── libmysqlclient.so.18
│ └── Other various files for linux
├── handler.py
├── requirements.py
├── requirements.txt
└── serverless.yml

Cross-compile as follows. (I just set the mount point to third-party /)

$ mkdir third-party
$ cd $_

$ docker run -it --rm -v "$PWD"/third-party/:/var/task "lambci/lambda:build-python2.7" bash

(same as above)

Then, python determines the OS and switches the loading destination. Insert third-party at the beginning of sys.path, and python doesn't seem to have LD_LIBRARY_PATH-like settings, so I'll load libmysqlclient.so.18 directly.

handler.py


import platform
if platform.system() != "Darwin":
    import sys
    import ctypes
    sys.path.insert(0, './third-party/')
    ctypes.CDLL("./third-party/libmysqlclient.so.18")

It's muddy, but I managed to achieve my goal.

After that, if you put the third-party directory in .gitignore and organize the cross-compilation process into a Dockerfile, it will be okay for configuration management.

Summary

Isn't it like this?

Recommended Posts

A story about cross-compiling a python package for AWS Lambda and deploying it serverless
About "Lamvery", a deployment and management tool for AWS Lambda
A story about Python pop and append
A story about modifying Python and adding functions
Build a local development environment for Lambda + Python using Serverless Framework
Install pip in Serverless Framework and AWS Lambda with Python environment
[Python] I wrote a REST API using AWS API Gateway and Lambda.
[AWS] Create a Python Lambda environment with CodeStar and do Hello World
[AWS lambda] Deploy including various libraries with lambda (generate a zip with a password and upload it to s3) @ Python
Touch AWS with Serverless Framework and Python
Periodically run a python program on AWS Lambda
Play with Lambda layer (python) for about 5 minutes
A story about Go's global variables and scope
A story about running Python on PHP on Heroku
Let's write a Python program and run it
Amazon API Gateway and AWS Lambda Python version
[For Python] Quickly create an upload file to AWS Lambda Layer
ImportError when trying to use gcloud package with AWS Lambda Python version
Connect to s3 with AWS Lambda Python
A story about cross-compiling a python package for AWS Lambda and deploying it serverless
[AWS / Lambda] How to load Python external library
Summary of studying Python to use AWS Lambda
How to use pip, a package management system that is indispensable for using Python
Try running a Schedule to start and stop an instance on AWS Lambda (Python)
Quickly create a Python data analysis dashboard with Streamlit and deploy it to AWS
Dynamic HTML pages made with AWS Lambda and Python
A story about making 3D space recognition with Python
Write about building a Python environment for writing Qiita Qiita
Building a Docker working environment for R and Python
A story about making Hanon-like sheet music with Python
A story about trying a (Golang +) Python monorepo with Bazel
A story about kindergartens, nursery schools, and children's gardens
Support for Python 2.7 runtime on AWS Lambda (as of 2020.1)
AWS Lambda now supports Python so I tried it
About creating and modifying custom themes for Python IDLE
Make ordinary tweets fleet-like with AWS Lambda and Python
A story that makes it easy to estimate the living area using Elasticsearch and Python
About Python for loops
A python lambda expression ...
About Python, for ~ (range)
Temporarily save a Python object and reuse it in another Python
AWS Amplify + API Gateway + Lambda + Python returns a normal response
A story about how to specify a relative path in python.
Associate Python Enum with a function and make it Callable
[Python] Create a date and time list for a specified period
Installation procedure for Python and Ansible with a specific version
Get a Python web page, character encode it, and display it
How to make a Python package (written for an intern)
A story about an amateur making a breakout with python (kivy) ②
A story about an amateur making a breakout with python (kivy) ①
Library for specifying a name server and dig with python
A story about trying to implement a private variable in Python.
Site monitoring and alert notification with AWS Lambda + Python + Slack
A story about a python beginner stuck with No module named'http.server'
The story that I set transparent proxy and it worked for some reason without a certificate
A story about everything from data collection to AI development and Web application release in Python (3. AI development)