Create a Layer for AWS Lambda Python with Docker

I wanted to register a module that I often use in the Python runtime of AWS Lambda with Lambda Layers, but I made it possible with Docker because it is troublesome to launch an EC2 instance and create a package.

What are Lambda Layers

--You can upload a zipped file and register it as a "layer". --Adding a layer to your Lambda Function will unpack the files under / opt at runtime and make them available at runtime. --If you put the module in the directory with the name specified for each language, it will pass the path so that it can be read from the Lambda Function. --For example, in the case of the Python runtime, if you put a Python module in a directory named python in the layer zip, you can load that module with ʻimport` on the Lambda Function side.

For more information: AWS Lambda Layers --AWS Lambda

Create a Docker image to create a layer

The Python 3.8 runtime seems to be running on Amazon Linux 2 [^ runtime], so you should normally set up an EC2 instance on Amazon Linux 2 and create layers in it (although Linux could be anything else). , At least layers made on macOS do not work on AWS Lambda [^ macos]), but in fact Amazon Linux has an official image distributed on Docker Hub [^ dockerhub], so install Python 3.8 based on this If you create a Docker image, you don't have to set up an EC2 instance.

[^ runtime]: AWS Lambda Runtime-AWS Lambda [^ macos]: Mac but I want to AWS Lambda with Python! --Qiita

Dockerfile

FROM amazonlinux:2

ARG PYTHON_VERSION=3.8.2

RUN yum update -y && yum install -y tar gzip make gcc openssl-devel bzip2-devel libffi-devel \
  && curl https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz | tar xz \
  && cd Python-${PYTHON_VERSION} && ./configure && make && make install \
  && cd - && rm -rf Python-${PYTHON_VERSION}

ADD entrypoint.sh /

RUN yum install -y zip \
  && mkdir /python \
  && chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

In Entrypoint, register a script to pip install and zip it.

entrypoint.sh

#!/bin/bash -eu

SRC=/python
DIST=/dist/layer.zip

pip3 install -t ${SRC} $@
rm -f ${DIST}
zip -q -r ${DIST} ${SRC}

Build the image.

$ docker build . -t bundle-pip-modules-for-aws-lambda-layers:python3.8

It seems that the image name is long, but anything is fine here.

Create a Layer with a Docker image

For example, if you want to create a Lambda Layer that can use Pandas.

$ docker run --rm \
    -v $(pwd):/dist \
    bundle-pip-modules-for-aws-lambda-layers:python3.8 \
        pandas

If successful, layer.zip will be created in the current directory.

In the above example, pandas was specified directly, but since all the arguments that can be specified by pip3 install can be passed for the arguments after the image name, for example, mount requirements.txt and then specify the -r option. You can also combine many modules into a layer with.

$ echo 'pandas' > requirements.txt
$ docker run --rm \
    -v $(pwd)/requirements.txt:/requirements.txt \
    -v $(pwd):/dist \
    bundle-pip-modules-for-aws-lambda-layers:python3.8 \
        -r requirements.txt

Register a layer

Upload the created layer.zip.

Use Layer

As a simplest example, let's create a Lambda Function that does nothing but load Pandas.

lambda_function.py

import pandas

def lambda_handler(event, context):
    pass

Add the registered Layer to the created Lambda Function.

If you try running this Lambda Function and you don't get an error, it's OK.