Make Lambda Layers with Lambda

Introduction

When I use Lambda, I usually use chalice to create and deploy Lambda functions.

chalice is convenient, but it seems that every time you deploy, you do pip install to collect the necessary package. Therefore, if you are using a heavier package such as pandas, it will take longer to deploy.

Therefore, use the LambdaLayers service of the Lambda that registers the package to be used in advance.

Premise

Prerequisites for creating LambdaLayers.

  1. For LambdaLayers, use the file that packages is zipped to make Lambda available in advance Reference /lambda/latest/dg/configuration-layers.html)
  2. The folder structure such as the layout of package is also decided.
  3. Some packages such as pandas need to be pip install on ʻAmazon Linux` reference
  4. It's easy to do on ʻEC2, but it's useless to create it if you don't have an environment such as VPC (VPC` itself costs money).
  5. Lambda has an area called / tmp that can be accessed by the program.

This condition (3) was so strict that Lambda sometimes did not recognize it properly with the zip file created locally. You can also do it on ʻEC2in (4), but if it is a private development that uses only serverless, There was a (my) request that I wanted to complete it with onlyLambda` as much as possible because I do not want to spend money.

Implementation

Flow of this implementation

Based on the above premise, this time create the zip file required for LambdaLayers by the following method.

ʻInstall`` packagein the/ tmp area of Lambda, zip it up with zip and upload it to S3 The actual registration ofLambda Layers` is done manually.

Source code

The program to be executed by Lambda is as follows.

lambda_function.py


import json
import subprocess
import zipfile
import boto3
from pathlib import Path


def lambda_handler(event, context):
    #Argument confirmation
    if "package_list" not in event:
        return {"error": "[package_list]Does not exist"}
    package_list = event['package_list']
    if type(package_list) is not list:
        return {"error": "[package_list]Is not a list"}    
    if "s3_bucket" not in event:
        return {"error": "[s3_bucket]Does not exist"}
    if "s3_key" not in event:
        return {"error": "[s3_key]Does not exist"}
    
    #Create a directory called python and install the package there
    mkdir_cmd = "mkdir -p /tmp/python"
    pip_install_cmd = f"pip install -t /tmp/python {' '.join(package_list)}"
    process = (subprocess.Popen(f"{mkdir_cmd} && {pip_install_cmd}", stdout=subprocess.PIPE, shell=True).communicate()[0]).decode('utf-8')
    print('commands...\n'+process)

    #Create Path object
    p = Path("/tmp/python")

    #Add the installed package to the zip
    with zipfile.ZipFile('/tmp/python_lib.zip', 'w', compression=zipfile.ZIP_DEFLATED) as new_zip:
        for f in list(p.glob("**/*")):
            new_zip.write(f, arcname=str(f).replace("/tmp/", ""))

    #Get S3 object and upload
    s3 = boto3.resource('s3')
    bucket = s3.Bucket(event['s3_bucket'])
    bucket.upload_file('/tmp/python_lib.zip', event['s3_key'])
    return {
        "success": f"s3://{event['s3_bucket']}/{event['s3_key']}Output to"
    }

Lambda settings

Please change it according to the package to be installed.

Argument example

When running, pass the following arguments to install the package contained in the package_list. It also uploads the zip file to the locations specified in s3_bucket and s3_key.

event


{
  "package_list": ["pandas", "lxml", "requests", "beautifulsoup4"],
  "s3_bucket": "<bucket name for s3>",
  "s3_key": "lambda_layer/python_lib.zip"
}

If you create a zip file using the above example, a zip file of about 43.1 [MB] will be created.

in conclusion

So, if you copy the above source code to Lambda and execute it, You can now create the zip required to register LambdaLayers.

Note: This time I have no choice but to run shell from Python. Basically, I don't think it's good to call shell from Python, so Let's use it when there is no choice but to use it.

Recommended Posts

Make Lambda Layers with Lambda
Make Yubaba with Discord.py
Make slides with iPython
Make ordinary tweets fleet-like with AWS Lambda and Python
Operate TwitterBot with Lambda, Python
Make sci-fi-like buttons with Kivy
Deploy Django serverless with Lambda
Easily write if-elif with lambda
Let's make Othello with wxPython
Easy to make with syntax
Make Puyo Puyo AI with Python
Let's make dice with tkinter
Make a fortune with Python
AWS Lambda with PyTorch [Lambda import]
Dynamically move Amplify with Lambda
Make a fire with kdeplot
Make Slack chatbot with Errbot
lambda
How to make AWS Lambda Layers when running selenium × chrome on AWS Lambda
Make Echolalia LINEbot with Python + heroku
AWS Lambda + Twilio Make Google Calendar reminders call with voice notifications
Make apache log csv with python
Make ASCII art with deep learning
[AWS] Create API with API Gateway + Lambda
Let's make a GUI with python.
Make a sound with Jupyter notebook
Face detection with Lambda (Python) + Rekognition
Let's make a breakout with wxPython
Let's make Othello AI with Chainer-Part 1-
Make testing with Selenium more accessible
[Python] Make the function a lambda function
Make a recommender system with python
Using Lambda with AWS Amplify with Go
Make compute server diskless with LTSP
Make a filter with a django template
[Lambda] Make import requests available [python]
Let's make a graph with python! !!
Let's make a supercomputer with xCAT
Notify HipChat with AWS Lambda (Python)
Let's make Othello AI with Chainer-Part 2-
Make a model iterator with PySide
Make people smile with Deep Learning
Make Scrapy an exe with Pyinstaller
Make a nice graph with plotly
Use PostgreSQL with Lambda (Python + psycopg2)
Let's make an IoT shirt with Lambda, Kinesis, Raspberry Pi [Part 1]
Try to make foldl and foldr with Python: lambda. Also time measurement
I tried to make "Sakurai-san" a LINE BOT with API Gateway + Lambda
Make the Python console covered with UNKO
[AWS] Using ini files with Lambda [Python]
Make an audio interface controller with pyusb (2)
Let's make a shiritori game with Python
Make a video player with PySimpleGUI + OpenCV
Make an audio interface controller with pyusb (1)
Make GUI apps super easy with tkinter
Make a rare gacha simulator with Flask
Make a Notebook Pipeline with Kedro + Papermill
Fractal to make and play with Python
Make a partially zoomed figure with matplotlib
Manipulate DynamoDB data with Lambda (Node & Python)
Make a drawing quiz with kivy + PyTorch