[AWS Lambda] Create a deployment package using the Docker image of Amazon Linux

If you want to use a library that is not in the AWS Lambda runtime environment, you need to download the library source group in the local environment, include it in the deployment package, and upload it to Lambda.

In the case of Python, for example, when creating a deployment package, follow the steps below.

$ virtualenv -p python2.7 venv
$ source venv/bin/activate
$ pip install -r requirements.txt
$ zip deploy_package.zip lambda_function.py #Zip the execution script
$ cd venv/lib/python2.7/site-packages
$ zip -r ../../../../deploy_package.zip * #Add libraries installed with pip to zip

If the library you want to use is pure Python code, this package will work on Lambda without any problem, but if the library depends on the function of the OS, Lambda is built and installed in the local environment. It doesn't work even if I upload it above. In the above example, pip install should be done in the same environment as Lambda's execution environment.

Failure example

Create and run a Lambda Function that uses a library called pycrypto.

requirements.txt


pycrypto==2.6.1

lambda_function.py


from Crypto.PublicKey import RSA


def lambda_handler(event, context):
    return {"messagge": "success"}

Prepare the above files and create a deployment package in your local OS X environment.

$ virtualenv -p python2.7 venv
$ source venv/bin/activate
$ pip install -r requirements.txt
$ zip deploy_package.zip lambda_function.py
$ cd venv/lib/python2.7/site-packages
$ zip -r ../../../../deploy_package.zip *

When I upload the created deploy_package.zip to AWS Lambda and execute it, I get the following error. Unable to import module 'lambda_function'

Unable to import module 'lambda_function'

Looking at the full text of the error log, it says ʻUnable to import module'lambda_function': /var/task/Crypto/Util/_counter.so: invalid ELF header. It seems that the header information of the file referenced by pycryptois invalid. The cause is that the environment wherepycrypto` is installed is different from the Lambda execution environment.

Countermeasures

You can work around this issue by installing the library using the Amazon Linux Docker image.

Prepare such a Dockerfile in the same hierarchy as the above file

Dockerfile


FROM amazonlinux:latest

RUN yum -y update && yum -y install gcc python27-devel
RUN cd /tmp && \
    curl https://bootstrap.pypa.io/get-pip.py | python2.7 - && \
    pip install virtualenv
WORKDIR /app
CMD virtualenv -p python2.7 venv-for-deployment && \
    source venv-for-deployment/bin/activate && \
    pip install -r requirements.txt

Running the command in this way creates a Python library code built on Amazon Linux with the name venv-for-deployment.

$ docker build . -t packager
$ docker run --rm -it -v $(PWD):/app packager

After that, create a zip of the deployment package as shown below and upload it to AWS Lambda.

$ zip deploy_package.zip lambda_function.py
$ cd venv-for-deployment/lib/python2.7/site-packages
$ zip -r ../../../../deploy_package.zip * .* #If dotfile is included".*"Also

When executed, the library can be imported and " success " is displayed safely.

success

Automation

Since there are a lot of commands to hit a little, it is convenient to create a Makefile like this because a zip will be generated just by make.

Makefile


package:
	docker build . -t packager
	docker run --rm -it -v $(PWD):/app packager
	zip deploy_package.zip lambda_function.py
	cd venv-for-deployment/lib/python2.7/site-packages && zip -r ../../../../deploy_package.zip * .*
	echo "Completed. Please upload deploy_package.zip to AWS Lambda"

sample

The sample Lambda Function used this time is in this repository. https://github.com/morishin/python-lambda-function-test

Impressions

AWS Lambda It's convenient, but when you try to do something a little elaborate, you almost cry.

Recommended Posts

[AWS Lambda] Create a deployment package using the Docker image of Amazon Linux
Create a Docker container image with JRE8 / JDK8 on Amazon Linux
Create a Layer for AWS Lambda Python with Docker
Avoiding the pitfalls of using a Mac (for Linux users?)
Create a GCE instance from a GCR Docker image using terraform
Generate a Docker image using Fabric
[Go] Create a CLI command to change the extension of the image
Using PhantomJS with AWS Lambda until displaying the html of the website
Since the dokcer image (1GB) of OpenJDK11 is large, create a small image (85MB) with alpine linux + jlink.
Create a graph using the Sympy module
A quick overview of the Linux kernel
[Linux] Build a Docker environment with Amazon Linux 2
I tried to create a model with the sample of Amazon SageMaker Autopilot
Put Python3 in Docker container of Amazon Linux2
I tried using the image filter of OpenCV
Create a GUI on the terminal using curses
Scraping the winning data of Numbers using Docker
I want to take a screenshot of the site on Docker using any font
Cut a part of the string using a Python slice
[CRUD] [Django] Create a CRUD site using the Python framework Django ~ 1 ~
Create Amazon Linux with AWS EC2 and log in
Create a QR code for the URL on Linux
Create a web page that runs a model that increases the resolution of the image using gradio, which makes it easy to create a web screen
[CRUD] [Django] Create a CRUD site using the Python framework Django ~ 2 ~
[Python] Mask the image into a circle using Pillow
[CRUD] [Django] Create a CRUD site using the Python framework Django ~ 3 ~
[CRUD] [Django] Create a CRUD site using the Python framework Django ~ 4 ~
[CRUD] [Django] Create a CRUD site using the Python framework Django ~ 5 ~
Create a shape on the trajectory of an object
Announcing the availability of Java 11 LTS on Amazon Linux 2
Create a dictionary by searching the table using sqlalchemy
[Linux] Create a self-signed certificate with Docker and apache
Example of using lambda
Change the Amazon Linux locale to Japan using Ansible's lineinfile
Instantly create a diagram of 2D data using python's matplotlib
Reuse the behavior of the @property method by using a descriptor [16/100]
Create a VS Code + Docker development environment on a Linux VM
Try a similar search for Image Search using the Python SDK [Search]
Image crawling summary performed at the speed of a second
Create a real-time auto-reply bot using the Twitter Streaming API
I made a dot picture of the image of Irasutoya. (part1)
A rough summary of the differences between Windows and Linux
Create API with Python, lambda, API Gateway quickly using AWS SAM
Create a correlation diagram from the conversation history of twitter
I made a dot picture of the image of Irasutoya. (part2)
[Linux] Write a deployment tool using rsync with a shell script
Create a gadget-like transparent background image type window using wxpython
Send a request from AWS Lambda to Amazon Elasticsearch Service
The story of creating a database using the Google Analytics API
[End of 2020] A memo to start using AWS CLI (Version 2)
Get the host name of the host PC with Docker on Linux
I wrote AWS Lambda, and I was a little addicted to the default value of Python arguments
Create a graph that displays an image with a mouse hover using the data visualization library Dash
Load an external library from your Lambda function using AWS Lambda Layers. The Python environment for Amazon Linux 2 is also in place. (Python3.6, Anaconda)