Upload what you got in request to S3 with AWS Lambda Python

At the women's club's Mokumokukai, I decided to make a crawler using AWS Lambda, so I lifted my back. The waist is really heavy.

Speaking of Lambda, it's free 1 million times a month, so I think you can run it 23 times a minute.

Crawler / scraping made with Lambda seems to be helpful, so at first I got it with request to S3 while looking at this I decided to upload something for the time being. The language is of course Python!

python


import boto3
import requests

BUCKET = 'test_requests'
s3 = boto3.client('s3')

def lambda_handler(event, context):
    key = 'corp.camon.tokyo'
    target_url = 'http://corp.camon.tokyo'
    target_html = requests.get(target_url).text
    s3.put_object(Bucket=BUCKET, Key=key, Body=target_html)

As expected, AWS Lambda is an easy win! When I press the Test button while thinking ...

python


"errorMessage": "Unable to import module 'lambda_function'"

Requests cannot be read and an error \ (^ o ^) / When I was wondering how to pip install on AWS Lambda, I came across this article.

Deploy AWS Lambda Python with lambda-uploader

python


pip install lambda-uploader

I made a directory, copied the source I wrote earlier, and prepared lambda.json and requirements.txt.

test_requests.py


import boto3
import requests

BUCKET = 'test_requests'
s3 = boto3.client('s3')

def lambda_handler(event, context):
    key = 'corp.camon.tokyo'
    target_url = 'http://corp.camon.tokyo'
    target_html = requests.get(target_url).text
    s3.put_object(Bucket=BUCKET, Key=key, Body=target_html)

lambda.json


  {
    "name": "test_requests",
    "description": "test requests",
    "region": "ap-northeast-1",
    "handler": "test_requests.lambda_handler",
    "role": "arn:aws:iam::????????????:role/lambda_s3_exec_role",
    "timeout": 300,
    "memory": 128
  }

requirements.txt


requests

The composition is like this.

python


$ tree
.
├── lambda.json
├── requirements.txt
└── test_requests.py

0 directories, 3 files

Now that it's ready, execute the command.

python


lambda-uploader

python


λ Building Package
λ Uploading Package
⁉️ Unexpected error. Please report this traceback.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/lambda_uploader/shell.py", line 151, in main
    _execute(args)
  File "/usr/local/lib/python2.7/site-packages/lambda_uploader/shell.py", line 82, in _execute
    upldr.upload(pkg)
  File "/usr/local/lib/python2.7/site-packages/lambda_uploader/uploader.py", line 112, in upload
    self.version = self.upload_new(pkg)
  File "/usr/local/lib/python2.7/site-packages/lambda_uploader/uploader.py", line 89, in upload_new
    Publish=self._config.publish,
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 301, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 398, in _make_api_call
    raise ClientError(parsed_response, operation_name)
ClientError: An error occurred (AccessDeniedException) when calling the CreateFunction operation: User: arn:aws:iam::????????????:user/WakanaYoshizawa is not authorized to perform: lambda:CreateFunction

If you don't have permission

スクリーンショット 2016-04-17 22.59.59.png

Add AWSLambdaFullAccess and run again

error··

At this point, the dots. Women's Club's Mokumokukai was imminently provided with beer at the end of the game, so I was sick and drank. I opened about half of the beer and got drunk, so I tried it with a slapstick ...

$ lambda-uploader
λ Building Package
λ Uploading Package
λ Fin

It seemed that it took time to reflect the policy attachment. It is important to develop with a margin.

I think I can do various things with Lambda, so I'll play with more \ (^ o ^) /

Recommended Posts

Upload what you got in request to S3 with AWS Lambda Python
Connect to s3 with AWS Lambda Python
[AWS] What to do when you want to pip with Lambda
[Python] Convert CSV file uploaded to S3 to JSON file with AWS Lambda
Upload files to Google Drive with Lambda (Python)
[AWS lambda] Deploy including various libraries with lambda (generate a zip with a password and upload it to s3) @ Python
I want to AWS Lambda with Python on Mac!
[Python] Scraping in AWS Lambda
What to do if you run python in IntelliJ and end with an error
[AWS] Try adding Python library to Layer with SAM + Lambda (Python)
Nice to meet you with python
Notify HipChat with AWS Lambda (Python)
Python | What you can do with Python
A note on what you did to use Flycheck with Python
What to do if you can't install pyaudio with pip #Python
What to do if you get a minus zero in Python
Send images taken with ESP32-WROOM-32 to AWS (API Gateway → Lambda → S3)
What I was addicted to with json.dumps in Python base64 encoding
[Python / AWS Lambda layers] I want to reuse only module in AWS Lambda Layers
Install pip in Serverless Framework and AWS Lambda with Python environment
[For Python] Quickly create an upload file to AWS Lambda Layer
Process the gzip file UNLOADed with Redshift with Python of Lambda, gzip it again and upload it to S3
If you are having trouble with timeouts when implementing Slack's SlashCommand in Python on AWS Lambda
[AWS] Using ini files with Lambda [Python]
Try logging in to qiita with Python
PUT gzip directly to S3 in Python
How to work with BigQuery in Python
[AWS] Link Lambda and S3 with boto3
To work with timestamp stations in Python
[AWS] Do SSI-like things with S3 / Lambda
Upload images to Google Drive with Python
Python + Selenium + Headless Chromium with aws lambda
ImportError when trying to use gcloud package with AWS Lambda Python version
What to do if you couldn't send an email to Yahoo with Python.
Easy server monitoring with AWS Lambda (Python) and result notification in Slack
What to do if you get lost in file reference with FileNotFoundError
What to do if you can't install with pip in babun environment
Move the turtle to the place where you click the mouse with turtle in Python
What to do if you get angry with "Value Error: unknown local: UTF-8" in python manage.py syncdb
How to upload with Heroku, Flask, Python, Git (4)
Upload scraped artifacts in Scrapy Cloud to S3
How to find out what kind of files are stored in S3 in Python
I was able to recurse in Python: lambda
You will be an engineer in 100 days --Day 35 --Python --What you can do with Python
LINE BOT with Python + AWS Lambda + API Gateway
Serverless application with AWS SAM! (APIGATEWAY + Lambda (Python))
Easily log in to AWS with multiple accounts
Sample to send slack notification with python lambda
What are you comparing with Python is and ==?
How to use tkinter with python in pyenv
What to do if you get an error when installing python with pyenv
What to do if you get "Python not configured." Using PyDev in Eclipse
[AWS / Lambda] How to load Python external library
[Python] What to check when you get a Unicode Decode Error in Django
Getting started with AWS IoT easily in Python
What Python beginners got hooked on with Django
Summary of studying Python to use AWS Lambda
Upload images to S3 with GUI using tkinter
Create a setting in terraform to send a message from AWS Lambda Python3.8 to Slack
What to do to get google spreadsheet in python
What are you using when testing with Python?