How to make AWS Lambda Layers when running selenium × chrome on AWS Lambda

In this article

When you want to operate the browser with selenium × chrome from AWS Lambda, I'm addicted to how to make a layer, so I'll post it.

Thing you want to do

I want to run selenium x chrome on AWS Lambda.

environment

windows 10 Pro python 3.7 chromdriver 2.37 headless-chromium 64.0.3282.167

Do not list

-How to create and call AWS lambda function -How to make a python program that can operate the browser with selenium webdriver.

Where I stumbled

  1. Unit for creating AWS Lambda Layers
  2. How to install and call AWS Lambda Layers

1. Unit for creating AWS Lambda Layers

First of all, AWS Lambda Layers are like common functions that can be used in common from lamda. It can be called in the form referenced from the Lambda main unit as shown below. By cutting out a part of the function, you can make the lambda body lighter. image.png By making it lighter, you can avoid the disadvantage that the code cannot be displayed because the capacity of the module set is too large, for example, as shown below. image.png

This time, I wanted to implement it with chromeDriver from python with selenium, so Create the following two layers.

** 1. Layer to store selenium library ** ** 2. Layer to store chromeDriver **

↓ Layer configuration image.png

The method of making is described below.

1. Layer to store selenium library

1. Preparation of the set of libraries to be installed

Execute the following command in any folder. I think you've already done pip install selenium, Since it is a task to prepare the library to be installed, it is done in a place different from the execution module.

Preparation of selenium module


   pip install -t ./python/lib/python3.7/site-packages selenium

2. The following configuration is completed. (Implemented in a windows environment.)

image.png

3. Zip compression

Zip it from the python folder. image.png

4. Create AWS Lambda Layers

Create a new Layers from the console screen of lambda. At the time of creation, specify the zip file of 3 and create it. image.png

5. Add Layer to your Lambda function

Reference the Layers created in 4 from the Lambda function. If you select ** Custom Layer ** from the Layer settings screen, it will be displayed in the options. image.png

6. Call with import

Call it from lambda with import.

import statement


from selenium import webdriver

If the above configuration is incorrect, the following error will occur.

error statement


[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named 'selenium'

2. Layer to store chromeDriver

1. Driver preparation

Prepare the officially distributed driver (2 points). chromdriver Distributor: https://chromedriver.storage.googleapis.com/index.html?path=2.37/ headless-chromium Distributor: https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-37/stable-headless-chromium-amazonlinux-2017-03.zip

2.zip compression

Place the above two points in the same folder and zip them in a linux environment. image.png

If you zip it in a windows environment, you will get the following error even if you run lamda. Even in the linux environment, if the permission of each file is not 777, the following error will occur.

error statement


[ERROR] WebDriverException: Message: 'chromedriver' executable may have wrong permissions. Please see https://sites.google.com/a/chromium.org/chromedriver/home

For windowsPC, there are the following methods to zip compress in linux environment. I could do either.

  1. Zip in WSL environment * Permission 777
  2. Create an appropriate environment with Docker desktop (I used this php environment that I happened to use. Https://hub.docker.com/_/php)

3. Add Layer to your Lambda function

Reference the Layers created in 4 from the Lambda function.

4. Call from lambda

Call it by specifying the following path from lambda. In the AWS Lambda layers specification, it is located in ** / opt **, so specify as follows.

import statement


driver = webdriver.Chrome(executable_path ="/opt/chromedriver", chrome_options=options)

If the path is incorrect, such as when there is no opt, an error will occur.

error statement


[ERROR] WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home

Another point to note

Both lamda and selenium are slow, so set the timeout value of lamda longer. The default is 3 seconds, so it almost times out and the following error occurs.

error statement


Task timed out after XX.XX seconds

Setting the timeout value image.png

Implementation

The final code for lamda is below.

lamda_function.py


from selenium import webdriver
from selenium.webdriver.chrome.options import Options

def lambda_handler(event, context):
    LINE_NOTIFY_URL = "https://notify-api.line.me/api/notify"

    options = Options()
    options.binary_location = '/opt/headless-chromium'
    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    options.add_argument('--single-process')
    options.add_argument('--disable-dev-shm-usage')

    driver = webdriver.Chrome(executable_path ='/opt/chromedriver', chrome_options=options)
    driver.get("https://xxxxxxxxxxx")

Summary

Looking at other articles, I found that most of them used serverless.yml or cloudFormation. I had little knowledge about that, so I took the above method. I hope it will be helpful for those who install selenium without a server for the first time.

Recommended Posts

How to make AWS Lambda Layers when running selenium × chrome on AWS Lambda
Things to note when running Python on EC2 from AWS Lambda
How to set layer on Lambda using AWS SAM
Summary of points I was addicted to running Selenium on AWS Lambda (python)
How to make Selenium as light as possible
[Python] Run Headless Chrome on AWS Lambda
Summary of how to write AWS Lambda
Tweet in Chama Slack Bot ~ How to make a Slack Bot using AWS Lambda ~
It is convenient to use Layers when putting a library on Lambda
[2020 version] How to install Python3 on AWS EC2
How to make multiple kernels selectable on Jupyter
How to deal with SessionNotCreatedException when using Selenium
[AWS / Lambda] How to load Python external library
How to make AWS rekognition recognize local image files
Make Lambda Layers with Lambda
How to debug selenium
Check types_map when using mimetypes on AWS Lambda (Python)
Try running a Schedule to start and stop an instance on AWS Lambda (Python)
How to download files from Selenium in Python in Chrome
I want to AWS Lambda with Python on Mac!
[AWS EC2] How to install Maven on Amazon Linux 2
A little trick to know when writing a Twilio application using Python on AWS Lambda
How to switch to smartphone mode with Python + Selenium + Chrome
How to operate Firefox with selenium on Windows Memo
[AWS] How to expand disk space on Amazon linux
Posted as an attachment to Slack on AWS Lambda (Python)
[AWS] What to do when you want to pip with Lambda
How to not load images when using PhantomJS with Selenium
How to make a multiplayer online action game on Slack
Addictive points when downloading files using boto on AWS Lambda
A quick explanation from creating AWS Lambda Layers to linking
[Python] Allow pip3 packages to be imported on AWS Lambda
How to register on pypi
How to use Python lambda
[Python / AWS Lambda layers] I want to reuse only module in AWS Lambda Layers
How to make a .dylib library from a .a library on OSX (El Capitan)
Slack --APIGateway --Lambda (Python) --How to make a RedShift interactive app
How to automatically install Chrome Driver for Chrome version with Python + Selenium + Chrome
How to create a serverless machine learning API with AWS Lambda
How to make a Japanese-English translation
How to install mysql-connector-python on mac
How to use Dataiku on Windows
Notes on how to use pywinauto
How to make a slack bot
How to install graph-tool on macOS
How to install VMware-Tools on Linux
How to install pycrypto on Windows
How to make a crawler --Advanced
How to deploy django-compressor on Windows
Run Python on Schedule on AWS Lambda
Notes on how to use featuretools
How to make a recursive function
[EC2] How to run selenium webdriver
How to add sudo when debugging
How to install OpenCV on Mac
How to run matplotlib on heroku
How to install PyPy on CentOS
How to use homebrew on Debian
Misunderstanding on how to connect cnn
How to install TensorFlow on CentOS 7
How to make a deadman's switch