[AWS / Lambda] How to load Python external library

Overview

AWS lambda cannot load external libraries such as pandas and numpy in the initial state. There are two ways to load it, the first is to upload the installed library and executable file to lambda as a whole in a zip file, and the second is to load it as a layer with lambda. The latter method is recommended, but in this article, we have summarized these methods as tips together with the aspect of code management.

environment

MacOS Catalina docker installed AWS CLI installed

Introduction

Since it is not good for code management to develop lambda on the browser, we will make it possible to develop locally. In other words, it takes the form of working locally so that it can be managed on gitHub and uploading code changes to lambda at any time. So I created a deployment environment using docker-lambda. (Please clone if you want to use it) https://github.com/shoda888/local2lambda

First, we need a docker image, so we will create it. (First time only) $ docker build -t shoda888/local2pylambda .

After that, if you give the lambda function name created in advance to the funcname variable of deploy.sh and execute the following, uploading will be done automatically. $ sh deploy.sh

Rewrite lambda_function.py and deploy.sh again to see if it has changed.

How to upload the library to lambda in a zip file

Just give requirements.txt the library name and deploy.sh (pandas, numpy, requests, etc.). A large number of library directories are created in the working directory. Pandas is too heavy and it takes about 5 minutes to upload. When I check it on lambda, the upload is successful, but a warning appears.

The deployment package for the Lambda function "*******" is too large to enable inline code editing. However, you can call the function.

スクリーンショット 2019-12-07 15.27.14.png

It seems that you can still call the function.

How to organize a library into a layer

It is recommended to leave requirements.txt empty as it is tedious to zip and upload the library every time. I think that it is convenient and reusable if the library is put together in a layer. (However, library version control may be troublesome in the future) Let's see how to organize the library into a layer.

For Pure Python libraries

In the case of a pure Python library, for example, a library composed of 100% Python as shown in the image can easily create a zip file for layer on your Mac without preparing Amazon Linux or ubuntu environment with EC2.  purepython.png

If you want to install ask_sdk_core as an example, this is all. (Example of python3.7)

$ mkdir -p build/python/lib/python3.7/site-packages
$ pip3 install ask_sdk_core -t build/python/lib/python3.7/site-packages/
$ cd build
$ zip -r ask_sdk.zip .

Add the generated ask_sdk.zip to layer and adapt the layer in lambda to load the library. スクリーンショット 2019-12-07 17.22.29.png layer.png

For non-Pure Python libraries

These include numpy and pandas. It seems that it is better to create an Amazon Linux environment with EC2 and zip it. (I felt that I made Layer with ubuntu before, but can ubuntu be used?)

Many people have already done this, so other people's articles will be very helpful. [Add pandas as a Lambda Layer] (https://qiita.com/thimi0412/items/4c725ec2b26aef59e5bd) Common libraries with AWS Lambda Layers

The following is youtube, but layer is made using ubuntu. (Easy to understand) [AWS Lambda Layers for Pandas library] (https://youtu.be/zrrH9nbSPhQ)

Summary

After making it possible to develop Lambda while managing the code locally, the method of loading the external library is summarized as Tips.

Recommended Posts

[AWS / Lambda] How to load Python external library
How to use Python lambda
How to use Requests (Python Library)
[Python] How to import the library
[AWS] Try adding Python library to Layer with SAM + Lambda (Python)
python3 How to install an external module
How to access RDS from Lambda (python)
Connect to s3 with AWS Lambda Python
Summary of how to write AWS Lambda
How to use the C library in Python
[2020 version] How to install Python3 on AWS EC2
How to install Python
How to use Python Image Library in python3 series
How to install python
Install python external library
Summary of studying Python to use AWS Lambda
How to set layer on Lambda using AWS SAM
[Python] How to use the graph creation library Altair
I want to AWS Lambda with Python on Mac!
[2020.8 latest] How to install Python
How to install Python [Windows]
python3: How to use bottle (2)
[Python] Scraping in AWS Lambda
[Python] How to use list 1
How to update Python Tkinter to 8.6
How to use Python argparse
Python: How to use pydub
How to change Python version
How to develop in Python
[python] How to judge scalar
[Python] How to use input ()
[Python] How to use virtualenv
python3: How to use bottle (3)
python3: How to use bottle
How to use Python bytes
How to debug the Python standard library in Visual Studio
[python] How to use the library Matplotlib for drawing graphs
How to launch AWS Batch from a python client app
How to execute external shell scripts and commands in python
Deploy external modules (numpy, scipy, requests, etc.) to AWS Lambda
[Python] Allow pip3 packages to be imported on AWS Lambda
How to get the information of organizations, Cost Explorer of another AWS account with Lambda (python)
How to install python using anaconda
How to write a Python class
[Python] How to FFT mp3 data
Python: How to use async with
Summary if using AWS Lambda (Python)
Introduction to Python Numerical Library NumPy
[Python] How to use Pandas Series
How to collect images in Python
How to use SQLite in Python
[Lambda] [Python] Post to Twitter from Lambda!
Write AWS Lambda function in Python
Run Python on Schedule on AWS Lambda
How to get the Python version
How to get started with Python
[Python] How to use list 3 Added
How to use Mysql in python
How to use OpenPose's Python API
[Python] How to swap array values
How to wrap C in Python