Things to note when running Python on EC2 from AWS Lambda

http://qiita.com/tf_qiita/items/90b4ec315d179b5d9b9c

Note that I solved the above problem.

Step 1: Specify the execution version

When using EC2 Run Command from Lambda, Run Command executes the version set in the initial state of the instance, not the Python executable ** that is enabled in the target EC2 instance. (For Amazon Linux, it is currently 2.7.12)

Therefore, when executing a version of Python that is different from the standard, specify the executable file with the full path in the Run Command described in Lambda as shown below. In the example below, the environment built in pyenv is specified. Since Lambda is executed as root, pyenv is built as root, but if you specify it directly and execute it, you should build a user for execution.

lambda.py


ssm.send_command(
    InstanceIds = instances,
    DocumentName = "AWS-RunShellScript",
    Parameters = {
        "commands": [
            "/root/.pyenv/versions/anaconda3-4.4.0/bin/python /work/hoge.py", 
        ],
        "executionTimeout": ["3600"]
    },
)

Step 2: Add library path

In step 1, the required version (environment) is executed without adding the path of Python to be executed to the environment variable, so the library cannot be imported as it is. Therefore, you can load the library by adding the site-packages path at the beginning of the script on the EC2 side called from Lambda. In the example below, site-packages under the environment specified in step 1 is specified.

ec2.py


import sys
sys.path.append("/root/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/")

Impressions

From the beginning, I assumed that it was a problem around the path, but it took time to isolate the cause on the Lambda side and the EC2 side. Also, is it unique to managed services, such as wasting effort in investigating whether the version is provided as a function? There was a failure. Similar low-layer stumbling blocks are foreseen in the future, but in that case I would like to recall this experience and return to the basics.

Recommended Posts

Things to note when running Python on EC2 from AWS Lambda
How to make AWS Lambda Layers when running selenium × chrome on AWS Lambda
[2020 version] How to install Python3 on AWS EC2
Modules cannot be imported in Python on EC2 run from AWS Lambda
Check types_map when using mimetypes on AWS Lambda (Python)
From python to running instance on google cloud platform
Things to note when initializing a list in Python
I want to AWS Lambda with Python on Mac!
Summary of points I was addicted to running Selenium on AWS Lambda (python)
Try running a Schedule to start and stop an instance on AWS Lambda (Python)
A little trick to know when writing a Twilio application using Python on AWS Lambda
Building an environment to execute python programs on AWS EC2
[Python] Allow pip3 packages to be imported on AWS Lambda
[Lambda] [Python] Post to Twitter from Lambda!
Run Python on Schedule on AWS Lambda
Update Python on Mac from 2 to 3
xgboost (python) on EC2 Spot instance environment prepared by AWS Lambda
Simple code to call a python program from Javascript on EC2
A note on using tab completion when running Python interactively on Windows
Everything from building a Python environment to running it on Windows
Change AWS EC2 instance from t2 to t3
Migrate Django applications running on Python 2.7 to Python 3.5
Connecting from python to MySQL on CentOS 6.4
How to access RDS from Lambda (python)
[Python] Run Headless Chrome on AWS Lambda
Connect to s3 with AWS Lambda Python
ImportError when trying to use gcloud package with AWS Lambda Python version
Things to watch out for when creating a Python environment on a Mac
When running a Python shell from Electron, pass multiple arguments to run Python.
How to start Python (Flask) when EC2 starts
Periodically run a python program on AWS Lambda
Points to note when switching from NAOqi OS 2.4.3 to 2.5.5
Python development on Ubuntu on AWS EC2 (using JupyterLab)
Note on encoding when LANG = C in Python
Python Note: When assigning a value to a string
[AWS / Lambda] How to load Python external library
Summary of studying Python to use AWS Lambda
What I did when updating from Python 2.6 to 2.7
I tried to reduce costs by starting / stopping EC2 collectively on AWS Lambda
Create a setting in terraform to send a message from AWS Lambda Python3.8 to Slack
A note I was addicted to when running Python with Visual Studio Code
Things to keep in mind when developing crawlers in Python
Changes from Python 3.0 to Python 3.5
Changes from Python 2 to Python 3.0
# 2 Build a Python environment on AWS EC2 instance (ubuntu18.04)
Terraform configured to launch AWS Lambda from Amazon SQS
Preferences to generate animated GIFs from Python on Mac
Things to keep in mind when copying Python lists
Python --Notes when converting from str type to int type
How to set layer on Lambda using AWS SAM
From buying a computer to running a program with python
Deploy Python3 function with Serverless Framework on AWS Lambda
Execute python3 system with PHP exec () on AWS EC2
Support for Python 2.7 runtime on AWS Lambda (as of 2020.1)
How to pass arguments when invoking python script from blender on the command line
Note to daemonize python
Tweet from AWS Lambda
From running MINST on TensorFlow 2.0 to visualization on TensorBoard (2019 edition)
Mercurial stopped working after migrating from Python 2 to Python 3 (Note)
[AWS EC2] How to install Maven on Amazon Linux 2
Things to keep in mind when processing strings in Python2