When I use Lambda
, I usually use chalice
to create and deploy Lambda
functions.
chalice
is convenient, but it seems that every time you deploy, you do pip install
to collect the necessary package
.
Therefore, if you are using a heavier package
such as pandas
, it will take longer to deploy.
Therefore, use the LambdaLayers
service of the Lambda
that registers the package
to be used in advance.
Prerequisites for creating LambdaLayers
.
LambdaLayers
, use the file that packages
is zipped to make Lambda
available in advance Reference /lambda/latest/dg/configuration-layers.html)package
is also decided.package
s such as pandas
need to be pip install
on ʻAmazon Linux` reference, but it's useless to create it if you don't have an environment such as
VPC (
VPC` itself costs money).Lambda
has an area called / tmp
that can be accessed by the program.This condition (3) was so strict that Lambda
sometimes did not recognize it properly with the zip
file created locally.
You can also do it on ʻEC2in (4), but if it is a private development that uses only serverless, There was a (my) request that I wanted to complete it with only
Lambda` as much as possible because I do not want to spend money.
Based on the above premise
, this time create the zip
file required for LambdaLayers
by the following method.
ʻInstall`` packagein the
/ tmp area of
Lambda, zip it up with
zip and upload it to
S3 The actual registration of
Lambda Layers` is done manually.
The program to be executed by Lambda
is as follows.
lambda_function.py
import json
import subprocess
import zipfile
import boto3
from pathlib import Path
def lambda_handler(event, context):
#Argument confirmation
if "package_list" not in event:
return {"error": "[package_list]Does not exist"}
package_list = event['package_list']
if type(package_list) is not list:
return {"error": "[package_list]Is not a list"}
if "s3_bucket" not in event:
return {"error": "[s3_bucket]Does not exist"}
if "s3_key" not in event:
return {"error": "[s3_key]Does not exist"}
#Create a directory called python and install the package there
mkdir_cmd = "mkdir -p /tmp/python"
pip_install_cmd = f"pip install -t /tmp/python {' '.join(package_list)}"
process = (subprocess.Popen(f"{mkdir_cmd} && {pip_install_cmd}", stdout=subprocess.PIPE, shell=True).communicate()[0]).decode('utf-8')
print('commands...\n'+process)
#Create Path object
p = Path("/tmp/python")
#Add the installed package to the zip
with zipfile.ZipFile('/tmp/python_lib.zip', 'w', compression=zipfile.ZIP_DEFLATED) as new_zip:
for f in list(p.glob("**/*")):
new_zip.write(f, arcname=str(f).replace("/tmp/", ""))
#Get S3 object and upload
s3 = boto3.resource('s3')
bucket = s3.Bucket(event['s3_bucket'])
bucket.upload_file('/tmp/python_lib.zip', event['s3_key'])
return {
"success": f"s3://{event['s3_bucket']}/{event['s3_key']}Output to"
}
Lambda
settingsPlease change it according to the package
to be installed.
512 [MB]
5[min]
S3
.When running, pass the following arguments to install the package
contained in the package_list
.
It also uploads the zip
file to the locations specified in s3_bucket
and s3_key
.
event
{
"package_list": ["pandas", "lxml", "requests", "beautifulsoup4"],
"s3_bucket": "<bucket name for s3>",
"s3_key": "lambda_layer/python_lib.zip"
}
If you create a zip
file using the above example, a zip
file of about 43.1 [MB]
will be created.
So, if you copy the above source code to Lambda
and execute it,
You can now create the zip
required to register LambdaLayers
.
Note: This time I have no choice but to run shell
from Python
.
Basically, I don't think it's good to call shell
from Python
, so
Let's use it when there is no choice but to use it.
Recommended Posts