[AWS] Try adding Python library to Layer with SAM + Lambda (Python)

SAM installation

Please refer to "[AWS] Try to create API Gateway + Lambda + DynamoDB sample with Serverless Application Model (SAM)".

As far as this sample is concerned, it is not necessary to have DynamoDB locally, so it is not necessary to install Docker.

Creating a project

Now, as usual, let's create a Hello World-based project.

$ sam init --runtime=python3.8
Which template source would you like to use?
	1 - AWS Quick Start Templates
	2 - Custom Template Location
Choice: 1

Project name [sam-app]:

Cloning app templates from https://github.com/awslabs/aws-sam-cli-app-templates.git

AWS quick start application templates:
	1 - Hello World Example
	2 - EventBridge Hello World
	3 - EventBridge App from scratch (100+ Event Schemas)
	4 - Step Functions Sample App (Stock Trader)
	5 - Elastic File System Sample App
Template selection: 1

-----------------------
Generating application:
-----------------------
Name: sam-app
Runtime: python3.8
Dependency Manager: pip
Application Template: hello-world
Output Directory: .

Next steps can be found in the README file at ./sam-app/README.md

Define Layer

First, create a directory for the layer. The directory is directly under the project, and the directory name will be described in the definition later, so you can freely add it. For the time being, let's call it "hello_world_layer".

$ mkdir hello_world_layer

Next, add the Layer definition under Resources in template.yaml. This time, create it with the name "Hello World Layer".

template.before changing yml


Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.8
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

template.After changing yml


Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.8
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get
      Layers:
        - !Ref HelloWorldLayer
  HelloWorldLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      Description: Layer description
      ContentUri: hello_world_layer/
      CompatibleRuntimes:
        - python3.8
    Metadata:
      BuildMethod: python3.8

Try adding NumPy

Then add the library. Please describe the library you want to use. In this sample, I would like to add NumPy, a numerical calculation module often used in machine learning.

Create requirements.txt in hello_world_layer and write as follows to add the library.

$ touch hello_world_layer/requirements.txt

hello_world_layer/requirements.txt


numpy

Create a Layer class

Let's create a user class that processes numpy under hello_world_layer. Should I name it appropriately as ʻuser_numpy`?

$ touch hello_world_layer/user_numpy.py

hello_world_layer/user_numpy.py


import numpy as np

class UserNumpy():
    def __init__(self):
        super().__init__()

    def array(self):
        return np.array([0,1,2])

Call a method of Layer class

Finally, let's call the added Layer method on the Function side. Let's change the existing hello_world / app.py as follows.

helloi_world/app.py


import json
from user_numpy import UserNumpy

def lambda_handler(event, context):
    un = UserNumpy()

    un_str = [str(n) for n in un.array()]
    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": ','.join(un_str)
        }),
    }

UserNumpy's array method returns the result of Numpy's ʻarray ([0,1,2])`. Since this is an array of numbers, after converting it to an array of character strings once, it is converted to a character string and returned as Json data.

Build

Let's build it.

$ sam build
Building function 'HelloWorldFunction'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
Building layer 'HelloWorldLayer'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

Deploy

Next is deployment. Please set the AWS credentials in advance.

$ sam deploy --guided

Configuring SAM deploy
======================

	Looking for samconfig.toml :  Found
	Reading default arguments  :  Success

	Setting default arguments for 'sam deploy'
	=========================================
	Stack Name [sam-app]:
	AWS Region [ap-northeast-1]:
	#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
	Confirm changes before deploy [Y/n]: y
	#SAM needs permission to be able to create roles to connect to the resources in your template
	Allow SAM CLI IAM role creation [Y/n]: y
	HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
	Save arguments to samconfig.toml [Y/n]: y

	Looking for resources needed for deployment: Not found.
	Creating the required resources...
	Successfully created!

		Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-fqjmg5s30i1w
		A different default S3 bucket can be set in samconfig.toml

	Deploying with following values
	===============================
	Stack name                 : sam-app
	Region                     : ap-northeast-1
	Confirm changeset          : True
	Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-fqjmg5s30i1w
	Capabilities               : ["CAPABILITY_IAM"]
	Parameter overrides        : {}

Initiating deployment
=====================

	Saved arguments to config file
	Running 'sam deploy' for future deployments will use the parameters saved above.
	The above parameters can be changed by modifying samconfig.toml
	Learn more about samconfig.toml syntax at
	https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html
Uploading to sam-app/ec3e7cb3e8873a2b56d81ae66ba15ec5  262144 / 538577.0  (48.67Uploading to sam-app/ec3e7cb3e8873a2b56d81ae66ba15ec5  524288 / 538577.0  (97.35Uploading to sam-app/ec3e7cb3e8873a2b56d81ae66ba15ec5  538577 / 538577.0  (100.00%)
Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  262144 / 14534393.0  (1.8Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  524288 / 14534393.0  (3.6Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  786432 / 14534393.0  (5.4Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  1048576 / 14534393.0  (7.Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  1310720 / 14534393.0  (9.Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  1572864 / 14534393.0  (10Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  1835008 / 14534393.0  (12Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  2097152 / 14534393.0  (14Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  2359296 / 14534393.0  (16Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  2621440 / 14534393.0  (18Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  2883584 / 14534393.0  (19Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  3145728 / 14534393.0  (21Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  3407872 / 14534393.0  (23Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  3670016 / 14534393.0  (25Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  3932160 / 14534393.0  (27Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  4194304 / 14534393.0  (28Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  4456448 / 14534393.0  (30Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  4718592 / 14534393.0  (32Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  4980736 / 14534393.0  (34Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  5242880 / 14534393.0  (36Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  5505024 / 14534393.0  (37Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  5767168 / 14534393.0  (39Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  6029312 / 14534393.0  (41Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  6145785 / 14534393.0  (42Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  6407929 / 14534393.0  (44Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  6670073 / 14534393.0  (45Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  6932217 / 14534393.0  (47Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  7194361 / 14534393.0  (49Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  7456505 / 14534393.0  (51Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  7718649 / 14534393.0  (53Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  7980793 / 14534393.0  (54Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  8242937 / 14534393.0  (56Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  8505081 / 14534393.0  (58Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  8767225 / 14534393.0  (60Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  9029369 / 14534393.0  (62Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  9291513 / 14534393.0  (63Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  9553657 / 14534393.0  (65Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  9815801 / 14534393.0  (67Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  10077945 / 14534393.0  (6Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  10340089 / 14534393.0  (7Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  10602233 / 14534393.0  (7Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  10864377 / 14534393.0  (7Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  11126521 / 14534393.0  (7Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  11388665 / 14534393.0  (7Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  11650809 / 14534393.0  (8Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  11912953 / 14534393.0  (8Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  12175097 / 14534393.0  (8Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  12437241 / 14534393.0  (8Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  12699385 / 14534393.0  (8Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  12961529 / 14534393.0  (8Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  13223673 / 14534393.0  (9Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  13485817 / 14534393.0  (9Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  13747961 / 14534393.0  (9Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  14010105 / 14534393.0  (9Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  14272249 / 14534393.0  (9Uploading to sam-app/04efcd43f42b7344dff7e02a5e9d321c  14534393 / 14534393.0  (100.00%)

	Deploying with following values
	===============================
	Stack name                 : sam-app
	Region                     : ap-northeast-1
	Confirm changeset          : True
	Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-fqjmg5s30i1w
	Capabilities               : ["CAPABILITY_IAM"]
	Parameter overrides        : {}

Initiating deployment
=====================
HelloWorldFunction may not have authorization defined.
Uploading to sam-app/5c2143d527c3bab330c5c2145019dcb1.template  1455 / 1455.0  (100.00%)

Waiting for changeset to be created..

CloudFormation stack changeset
------------------------------------------------------------------------------------------------
Operation                        LogicalResourceId                ResourceType
------------------------------------------------------------------------------------------------
+ Add                            HelloWorldFunctionHelloWorldPe   AWS::Lambda::Permission
                                 rmissionProd
+ Add                            HelloWorldFunctionRole           AWS::IAM::Role
+ Add                            HelloWorldFunction               AWS::Lambda::Function
+ Add                            HelloWorldLayer8a10103d68        AWS::Lambda::LayerVersion
+ Add                            ServerlessRestApiDeployment47f   AWS::ApiGateway::Deployment
                                 c2d5f9d
+ Add                            ServerlessRestApiProdStage       AWS::ApiGateway::Stage
+ Add                            ServerlessRestApi                AWS::ApiGateway::RestApi
------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:ap-northeast-1:767054379442:changeSet/samcli-deploy1597147509/8e325896-aa90-4379-afa2-44147e907291


Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y

2020-08-11 21:05:21 - Waiting for stack create/update to complete

CloudFormation events from changeset
-------------------------------------------------------------------------------------------------
ResourceStatus           ResourceType             LogicalResourceId        ResourceStatusReason
-------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS       AWS::Lambda::LayerVers   HelloWorldLayer8a10103   -
                         ion                      d68
CREATE_IN_PROGRESS       AWS::IAM::Role           HelloWorldFunctionRole   Resource creation
                                                                           Initiated
CREATE_IN_PROGRESS       AWS::IAM::Role           HelloWorldFunctionRole   -
CREATE_IN_PROGRESS       AWS::Lambda::LayerVers   HelloWorldLayer8a10103   Resource creation
                         ion                      d68                      Initiated
CREATE_COMPLETE          AWS::Lambda::LayerVers   HelloWorldLayer8a10103   -
                         ion                      d68
CREATE_COMPLETE          AWS::IAM::Role           HelloWorldFunctionRole   -
CREATE_IN_PROGRESS       AWS::Lambda::Function    HelloWorldFunction       -
CREATE_IN_PROGRESS       AWS::Lambda::Function    HelloWorldFunction       Resource creation
                                                                           Initiated
CREATE_COMPLETE          AWS::Lambda::Function    HelloWorldFunction       -
CREATE_IN_PROGRESS       AWS::ApiGateway::RestA   ServerlessRestApi        -
                         pi
CREATE_IN_PROGRESS       AWS::ApiGateway::RestA   ServerlessRestApi        Resource creation
                         pi                                                Initiated
CREATE_COMPLETE          AWS::ApiGateway::RestA   ServerlessRestApi        -
                         pi
CREATE_IN_PROGRESS       AWS::ApiGateway::Deplo   ServerlessRestApiDeplo   -
                         yment                    yment47fc2d5f9d
CREATE_IN_PROGRESS       AWS::Lambda::Permissio   HelloWorldFunctionHell   Resource creation
                         n                        oWorldPermissionProd     Initiated
CREATE_IN_PROGRESS       AWS::Lambda::Permissio   HelloWorldFunctionHell   -
                         n                        oWorldPermissionProd
CREATE_COMPLETE          AWS::ApiGateway::Deplo   ServerlessRestApiDeplo   -
                         yment                    yment47fc2d5f9d
CREATE_IN_PROGRESS       AWS::ApiGateway::Deplo   ServerlessRestApiDeplo   Resource creation
                         yment                    yment47fc2d5f9d          Initiated
CREATE_IN_PROGRESS       AWS::ApiGateway::Stage   ServerlessRestApiProdS   -
                                                  tage
CREATE_IN_PROGRESS       AWS::ApiGateway::Stage   ServerlessRestApiProdS   Resource creation
                                                  tage                     Initiated
CREATE_COMPLETE          AWS::ApiGateway::Stage   ServerlessRestApiProdS   -
                                                  tage
CREATE_COMPLETE          AWS::Lambda::Permissio   HelloWorldFunctionHell   -
                         n                        oWorldPermissionProd
CREATE_COMPLETE          AWS::CloudFormation::S   sam-app                  -
                         tack
-------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
-------------------------------------------------------------------------------------------------
Outputs
-------------------------------------------------------------------------------------------------
Key                 HelloWorldFunctionIamRole
Description         Implicit IAM Role created for Hello World function
Value               arn:aws:iam::767054379442:role/sam-app-HelloWorldFunctionRole-9AHTS9BZMUQL

Key                 HelloWorldApi
Description         API Gateway endpoint URL for Prod stage for Hello World function
Value               https://co2snvo0lk.execute-api.ap-northeast-1.amazonaws.com/Prod/hello/

Key                 HelloWorldFunction
Description         Hello World Lambda Function ARN
Value               arn:aws:lambda:ap-northeast-1:767054379442:function:sam-app-
HelloWorldFunction-1GVL3Y25TQYGZ
-------------------------------------------------------------------------------------------------

Successfully created/updated stack - sam-app in ap-northeast-1

Run

Finally, let's access the API Gateway endpoint that is output during Deploy.

$ curl https://co2snvo0lk.execute-api.ap-northeast-1.amazonaws.com/Prod/hello/
{"message": "0,1,2"}

You can see that the result of ʻarray ([0,1,2])` is returned as a string in the response! !!

Summary

You've found it easy to add a library to a Layer and create a common module that calls it. In the case of serverless, it is desirable to design it as loosely coupled as possible, but I think that it is not a mistake to place common logic within the same domain in the layer.

However, as a Lambda constraint

--Deployment package size --Compressed and 50MB --250MB after decompression

Since there is a restriction, let's take this into consideration and import only the necessary libraries.

Recommended Posts

[AWS] Try adding Python library to Layer with SAM + Lambda (Python)
Connect to s3 with AWS Lambda Python
Serverless application with AWS SAM! (APIGATEWAY + Lambda (Python))
[AWS / Lambda] How to load Python external library
How to set layer on Lambda using AWS SAM
Create a Layer for AWS Lambda Python with Docker
I want to AWS Lambda with Python on Mac!
Try to operate Facebook with Python
Create API with Python, lambda, API Gateway quickly using AWS SAM
Notify HipChat with AWS Lambda (Python)
[AWS SAM] Introduction to Python version
[Python] Convert CSV file uploaded to S3 to JSON file with AWS Lambda
[SAM] Try using RDS Proxy with Lambda (Python) [user/pass, IAM authentication]
Try adding a wall to your IFC file with IfcOpenShell python
[For Python] Quickly create an upload file to AWS Lambda Layer
[AWS] Using ini files with Lambda [Python]
Try to reproduce color film with Python
I want to play with aws with python
Try HTML scraping with a Python library
Try assigning or switching with Python: lambda
Python + Selenium + Headless Chromium with aws lambda
ImportError when trying to use gcloud package with AWS Lambda Python version
Try to make foldl and foldr with Python: lambda. Also time measurement
[Cloudian # 1] Try to access object storage with AWS SDK for Python (boto3)
Upload what you got in request to S3 with AWS Lambda Python
[AWS SAM] Create API with DynamoDB + Lambda + API Gateway
Library comparison summary to generate PDF with Python
Try to solve the man-machine chart with Python
Try to draw a life curve with python
Play with Lambda layer (python) for about 5 minutes
Try to make a "cryptanalysis" cipher with Python
LINE BOT with Python + AWS Lambda + API Gateway
Try to automatically generate Python documents with Sphinx
Sample to send slack notification with python lambda
Manage AWS nicely with the Python library Boto
[AWS] Try tracing API Gateway + Lambda with X-Ray
Python mock to try AWS IoT Device Shadow
Export RDS snapshot to S3 with Lambda (Python)
Try to make a dihedral group with Python
Upload files to Google Drive with Lambda (Python)
Summary of studying Python to use AWS Lambda
Try to detect fish with python + OpenCV2.4 (unfinished)
Try implementing a Cisco Spark bot with AWS Lambda + Amazon API Gateway (Python)
Feel free to turn Python using the library into an AWS Lambda function
Try scraping with Python.
AWS CDK with Python
Try AWS Lambda Destinations
Try to solve the programming challenge book with python3
Dynamic HTML pages made with AWS Lambda and Python
[First API] Try to get Qiita articles with Python
Python beginners try adding basic auth to Django admin
Try to make a command standby tool with python
[AWS] Play with Step Functions (SAM + Lambda) Part.3 (Branch)
Try to solve the internship assignment problem with Python
Create Python version Lambda function (+ Lambda Layer) with Serverless Framework
Try to operate DB with Python and visualize with d3
Deploy Python3 function with Serverless Framework on AWS Lambda
Write multiple records to DynamoDB with Lambda (Python, JavaScript)
[AWS] Play with Step Functions (SAM + Lambda) Part.1 (Basic)
Try to automate pdf format report creation with Python
Make ordinary tweets fleet-like with AWS Lambda and Python