At the women's club's Mokumokukai, I decided to make a crawler using AWS Lambda, so I lifted my back. The waist is really heavy.
Speaking of Lambda, it's free 1 million times a month, so I think you can run it 23 times a minute.
Crawler / scraping made with Lambda seems to be helpful, so at first I got it with request
to S3 while looking at this I decided to upload something for the time being.
The language is of course Python!
python
import boto3
import requests
BUCKET = 'test_requests'
s3 = boto3.client('s3')
def lambda_handler(event, context):
key = 'corp.camon.tokyo'
target_url = 'http://corp.camon.tokyo'
target_html = requests.get(target_url).text
s3.put_object(Bucket=BUCKET, Key=key, Body=target_html)
As expected, AWS Lambda is an easy win! When I press the Test button while thinking ...
python
"errorMessage": "Unable to import module 'lambda_function'"
Requests
cannot be read and an error \ (^ o ^) /
When I was wondering how to pip install
on AWS Lambda, I came across this article.
Deploy AWS Lambda Python with lambda-uploader
python
pip install lambda-uploader
I made a directory, copied the source I wrote earlier, and prepared lambda.json and requirements.txt.
test_requests.py
import boto3
import requests
BUCKET = 'test_requests'
s3 = boto3.client('s3')
def lambda_handler(event, context):
key = 'corp.camon.tokyo'
target_url = 'http://corp.camon.tokyo'
target_html = requests.get(target_url).text
s3.put_object(Bucket=BUCKET, Key=key, Body=target_html)
lambda.json
{
"name": "test_requests",
"description": "test requests",
"region": "ap-northeast-1",
"handler": "test_requests.lambda_handler",
"role": "arn:aws:iam::????????????:role/lambda_s3_exec_role",
"timeout": 300,
"memory": 128
}
requirements.txt
requests
The composition is like this.
python
$ tree
.
├── lambda.json
├── requirements.txt
└── test_requests.py
0 directories, 3 files
Now that it's ready, execute the command.
python
lambda-uploader
python
λ Building Package
λ Uploading Package
⁉️ Unexpected error. Please report this traceback.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/lambda_uploader/shell.py", line 151, in main
_execute(args)
File "/usr/local/lib/python2.7/site-packages/lambda_uploader/shell.py", line 82, in _execute
upldr.upload(pkg)
File "/usr/local/lib/python2.7/site-packages/lambda_uploader/uploader.py", line 112, in upload
self.version = self.upload_new(pkg)
File "/usr/local/lib/python2.7/site-packages/lambda_uploader/uploader.py", line 89, in upload_new
Publish=self._config.publish,
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 301, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 398, in _make_api_call
raise ClientError(parsed_response, operation_name)
ClientError: An error occurred (AccessDeniedException) when calling the CreateFunction operation: User: arn:aws:iam::????????????:user/WakanaYoshizawa is not authorized to perform: lambda:CreateFunction
If you don't have permission
Add AWSLambdaFullAccess and run again
error··
At this point, the dots. Women's Club's Mokumokukai was imminently provided with beer at the end of the game, so I was sick and drank. I opened about half of the beer and got drunk, so I tried it with a slapstick ...
$ lambda-uploader
λ Building Package
λ Uploading Package
λ Fin
It seemed that it took time to reflect the policy attachment. It is important to develop with a margin.
I think I can do various things with Lambda, so I'll play with more \ (^ o ^) /
Recommended Posts