[DOCKER] I want to AWS Lambda with Python on Mac!

I heard that AWS Lambda supports Python 3.6, so I decided to write a Lambda Function written in Node.js in Python, but Lambda including a module written in C such as numpy When I deploy a Function from a Mac, it's hard to run it on Lambda because it says ** "It's a module built for Mac so I can't load it" **.

I solved this problem with Docker, so I will summarize how to do it.

[Added on 2020.03.25] I also wrote about how to create a Lambda Layer. Create a Layer for AWS Lambda Python with Docker-Qiita

An example of a Lambda Function you want to deploy

This time, consider deploying a Lambda Function that does nothing but load numpy like this:

Directory structure


.
├── main.py
└── requirements.txt

main.py


import numpy

def handler(event, context):
    pass

requirements.txt


numpy

When installed normally on a Mac

The following command installs numpy in the same directory, but even if I zip and deploy the installed one, it doesn't work on Lambda because numpy is a build for Mac.

$ pip install -r requirements.txt -t .

You should probably get this error.

Unable to import module 'main':
Importing the multiarray numpy extension module failed. Most likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all files not under version control). Otherwise reinstall numpy.

Original error was: cannot import name 'multiarray'

When I google the solution, I get only a mysterious answer like "Launch an Amazon Linux instance and build it there". ** I want to be serverless with AWS Lambda, but I need a server for deployment ** How is this? ?? ??

In short, I wish I could build in a Linux environment. You can do that with Docker.

Build on Docker container

To execute the above command on the Docker container, do as follows.

$ docker run --rm -v $(pwd):/work -w /work python:3.6 pip install -r requirements.txt -t .

Start the container from the Python 3.6 image [^ linux], mount the current directory in the working directory / work, and install the package there. When the build is complete, this container is no longer needed, so I added the --rm option to remove it.

[^ linux]: python: 3.6 The image is based on Debian, so it is different from the Lambda execution environment (Amazon Linux), but it was installed numpy-1.13.1-cp36-cp36m-manylinux1_x86_64.whl So I think Linux is fine. There may be some bad modules other than numpy, but in that case you can drop the Amazon Linux Docker image.

Now that you have numpy built in a Linux environment, you should be able to solidify and deploy it.

How to develop

You might wonder, "Once you've installed a module for Linux, you can't run it on your Mac during development."

Sure, it can't run on a Mac, but I think it's best to run it all in a Docker container when developing.

$ docker run --rm -v $(pwd):/work -w /work python:3.6 python -c 'import main; main.handler({}, None)'

Like this.

Recommended Posts

I want to AWS Lambda with Python on Mac!
I want to play with aws with python
I want to debug with Python
I want to use Linux on mac
I want to analyze logs with Python
Connect to s3 with AWS Lambda Python
I want to know if you install Python on Mac ・ Iroha
[Python / AWS Lambda layers] I want to reuse only module in AWS Lambda Layers
I want to use MATLAB feval with python
I want to make a game with Python
I want to use Temporary Directory with Python2
I want to solve APG4b with Python (Chapter 2)
I want to write to a file with Python
I want to set up a GUI development environment with Python or Golang on Mac
I want to do it with Python lambda Django, but I will stop
I want to tweet on Twitter with Python, but I'm addicted to it
I want to handle optimization with python and cplex
I tried to implement Minesweeper on terminal with python
I want to inherit to the back with python dataclass
I want to work with a robot in python.
Deploy Python3 function with Serverless Framework on AWS Lambda
[ML Ops] I want to do multi-project with Python
I want to run a quantum computer with Python
I want to bind a local variable with lambda
Summary of points I was addicted to running Selenium on AWS Lambda (python)
Run Python on Schedule on AWS Lambda
I want to do ○○ with Pandas
Notify HipChat with AWS Lambda (Python)
Steps to install python3 on mac
Update Python on Mac from 2 to 3
Posted as an attachment to Slack on AWS Lambda (Python)
I want to be able to analyze data with Python (Part 3)
[AWS] What to do when you want to pip with Lambda
I want to specify another version of Python with pyvenv
[AWS] Try adding Python library to Layer with SAM + Lambda (Python)
Challenge problem 5 with Python: lambda ... I decided to copy without
I want to be able to analyze data with Python (Part 4)
I want to be able to analyze data with Python (Part 2)
I want to automatically attend online classes with Python + Selenium!
[Python] I want to use the -h option with argparse
[Python] Allow pip3 packages to be imported on AWS Lambda
[Mac] I want to make a simple HTTP server that runs CGI with Python
I want to detect objects with OpenCV
[Python] Convert CSV file uploaded to S3 to JSON file with AWS Lambda
I want to use a wildcard that I want to shell with Python remove
I want to know the weather with LINE bot feat.Heroku + Python
I want to monitor UNIQLO + J page updates [Scraping with python]
I want to solve APG4b with Python (only 4.01 and 4.04 in Chapter 4)
I want to blog with Jupyter Notebook
I want to output the beginning of the next month with Python
I want to use jar from python
I wanted to solve ABC160 with Python
Things to note when running Python on EC2 from AWS Lambda
Follow active applications on Mac with Python
I want to build a Python environment
I want to pip install with PythonAnywhere
How to erase Python 2.x on Mac.
Build Python environment with Anaconda on Mac
python on mac
I tried to display GUI on Mac with X Window System
[Introduction] I want to make a Mastodon Bot with Python! 【Beginners】