I just built a virtual environment with AWS lambda layer

Background I used to save the pip package using venv in Python3.6 or later, but when I was investigating whether it could be done on AWS, there was an affordable function called layer, so I tried using it.

venv

This is OK on the work PC

python -m venv [Virtual environment name]

./[Virtual environment name]/bin/activate
([Virtual environment name]) pip install [package name]

Zip for package

[Web scraping using AWS lambda-Development (Web scraping)](https://qiita.com/satsukiya/items/b9d02abd7fa96cd59355#development-web%E3%82%B9%E3%82%AF%E3% Create a zip file for uploading on AWS in the same way as 83% AC% E3% 82% A4% E3% 83% 94% E3% 83% B3% E3% 82% B0).

mkdir packages
cd packages
pip install [package] -t ./
pip install [package] -t ./
......
zip -r ./myDeploymentPackage.zip ./packages

Layer To create a pip environment, create a layer.

Development (error)

Here, [Web scraping using AWS lambda-Development (Web scraping)](https://qiita.com/satsukiya/items/b9d02abd7fa96cd59355#development-web%E3%82%B9%E3%82%AF% Let's write a code using beautiful soup with E3% 83% AC% E3% 82% A4% E3% 83% 94% E3% 83% B3% E3% 82% B0).

import json
import requests
from bs4 import BeautifulSoup

def lambda_handler(event, context):
    # TODO implement

    response = requests.get('https://mainichi.jp/editorial/')
    soup = BeautifulSoup(response.text)
    pages = soup.find("ul", class_="list-typeD")
    articles = pages.find_all("article")
    links = [ "https:" + a.a.get("href") for a in articles]

    return {
        'statusCode': 200,
        'links' : links
    }

Then

{
  "errorMessage": "Unable to import module 'lambda_function'"
}

I get an import error.

[Basic precautions when using Lambda @ Python layers (for those who cannot import files uploaded to layers)-Notes](https://qiita.com/k_hoso/items/78beb33e53abfdddabe7#%E6% B3% A8% E6% 84% 8F% E7% 82% B9) Looking at, it seems that the zip file is decompressed under / opt /.

In this case, there is a package installed by pip on the work PC in / opt / packages /, so

import sys
sys.path.append('/opt/packages')

It is necessary to pass the path in advance.

Development (modified)

import sys
sys.path.append('/opt/packages')
import json
import requests
from bs4 import BeautifulSoup


def lambda_handler(event, context):
    # TODO implement

    response = requests.get('https://mainichi.jp/editorial/')
    soup = BeautifulSoup(response.text)
    pages = soup.find("ul", class_="list-typeD")
    articles = pages.find_all("article")
    links = [ "https:" + a.a.get("href") for a in articles]

    return {
        'statusCode': 200,
        'links' : links
    }
{
  "statusCode": 200,
  "links": [
    "https://mainichi.jp/articles/20201120/ddm/005/070/067000c",
    "https://mainichi.jp/articles/20201120/ddm/005/070/065000c",
    "https://mainichi.jp/articles/20201119/ddm/005/070/119000c",
    "https://mainichi.jp/articles/20201119/ddm/005/070/118000c",
    "https://mainichi.jp/articles/20201118/ddm/005/070/115000c",
    "https://mainichi.jp/articles/20201118/ddm/005/070/114000c",
    "https://mainichi.jp/articles/20201117/ddm/005/070/114000c",
    "https://mainichi.jp/articles/20201117/ddm/005/070/113000c",
    "https://mainichi.jp/articles/20201116/ddm/005/070/043000c"
  ]
}

Post Scripting I wish I could do it without "/ packages /" when compressing with zip: expressionless:

Recommended Posts

I just built a virtual environment with AWS lambda layer
I just did FizzBuzz with AWS Lambda
Create a virtual environment with Python!
Building a virtual environment with Python 3
I want to use a virtual environment with jupyter notebook!
I wrote a script to create a Twitter Bot development environment quickly with AWS Lambda + Python 2.7
[Python] Create a virtual environment with Anaconda
Create a virtual environment with Python_Mac version
I built a TensorFlow environment on windows10
Build a python virtual environment with pyenv
I started Node.js in a virtual environment
[AWS] Create a Python Lambda environment with CodeStar and do Hello World
I wrote a Slack bot that notifies delay information with AWS Lambda
Build a WardPress environment on AWS with pulumi
Start Django in a virtual environment with Pipenv
Create a virtual environment with conda in Python
[Memo] Build a virtual environment with Pyenv + anaconda
Build a virtual environment with pyenv and venv
Work in a virtual environment with Python virtualenv.
Flow of creating a virtual environment with Anaconda
I tried connecting AWS Lambda with other services
When I tried to create a virtual environment with Python, it didn't work
Virtual environment with Python 3.6
[AWS] I made a reminder BOT with LINE WORKS
Create a virtual environment with Anaconda installed via Pyenv
code-server Online environment (2) Create a virtual network with Boto3
I made a Python3 environment on Ubuntu with direnv.
I tried running TensorFlow in AWS Lambda environment: Preparation
Build a python virtual environment with virtualenv and virtualenvwrapper
I want to AWS Lambda with Python on Mac!
Notes on creating a virtual environment with Anaconda Navigator
Build a Django environment for Win10 (with virtual space)
I want to bind a local variable with lambda
I built an AWS Chalice development environment with docker and tried deploying a serverless application at super high speed
[AWS] Try adding Python library to Layer with SAM + Lambda (Python)
I want to start a jupyter environment with one command
Building a Python virtual environment
AWS Lambda with PyTorch [Lambda import]
Switch virtual environment with jupyter
Building a Python virtual environment
Building a virtual environment with pyenv-virtualenv/Python (installation, environment settings, packages) Mac environment
[AWS] I made a reminder BOT with LINE WORKS (implementation)
A story that I was addicted to calling Lambda from AWS Lambda.
Build a Flask / Bottle-like web application on AWS Lambda with Chalice
I made a Twitter Bot with Go x Qiita API x Lambda
Serverless scraping on a regular basis with AWS lambda + scrapy Part 1
Prepare an environment to use OpenCV and Pillow with AWS Lambda
Install pip in Serverless Framework and AWS Lambda with Python environment
I made a development environment for Django 3.0 with Docker, Docker-compose, Poetry
Postgres environment construction with Docker I struggled a little, so note
[Python] I wrote a REST API using AWS API Gateway and Lambda.
How to create a serverless machine learning API with AWS Lambda
Building a kubernetes environment with ansible 2
Activate Anaconda's virtual environment with PowerShell
Touch AWS Lambda environment variable support
Build python virtual environment with virtualenv
ruby environment construction with aws EC2
[AWS] Create API with API Gateway + Lambda
I made a fortune with Python.
Try giving AWS Lambda environment variables?
Using Lambda with AWS Amplify with Go