[AWS] Link Lambda and S3 with boto3

Introduction

Create a temporary file with AWS Lambda and upload it to S3. In addition, I will try to download the file uploaded to S3 to the original location.

System configuration

Lambda_S3.png

  1. Create a file in / tmp / log / sess / and archive the directory in /tmp/sess-info.tar.
  2. Upload sess-info.tar to S3 \ .
  3. Delete /tmp/sess-info.tar and download sess-info.tar from S3 \ to /tmp/sess-info.tar.
  4. Delete / tmp / log / sess / if it exists and expand sess-info.tar to / tmp / log / sess /.

Lambda Code

import boto3
import os
import os.path
import tarfile
import shutil

def lambda_handler(event, context):
    bucket_name = 'bucket_name'
    tmp_dir = '/tmp/'
    log_dir = '/tmp/log/sess/'
    file_name = 'sess-info.tar'

    key = file_name
    file_path = tmp_dir + file_name

    # === Step 1
    # create log dir
    if os.path.exists(log_dir) == False:
        os.makedirs(log_dir)

    # write log
    with open(log_dir + 'log1.txt', 'w') as file:
        file.write('hoge\n')
    with open(log_dir + 'log2.txt', 'w') as file:
        file.write('fuga\n')

    # create log archive
    with tarfile.open(file_path, mode='w:gz') as archive:
        archive.add(log_dir)

    # === Step 2
    # create s3 resource
    s3 = boto3.resource('s3')
    bucket = s3.Bucket(bucket_name)

    # upload log archive
    bucket.upload_file(file_path, key)

    # === Step 3
    # delete log archive
    if os.path.exists(file_path):
        os.remove(file_path)

    # download log archive
    bucket.download_file(key, file_path)

    # === Step 4
    # delete log dir
    if os.path.exists(log_dir):
        shutil.rmtree(log_dir)

    # extract log archive
    with tarfile.open(file_path, mode='r:gz') as archive:
        archive.extractall('/')

Lambda Configuration

S3 Bucket Create a bucket with the same name as the Lambda Code bucket_name.

in conclusion

Lambda is basically designed stateless, but there are times when you want to access stateful data. In such a case, we will take advantage of the following.

This time, it was an example of using Lambda's / tmp directory and S3.

Recommended Posts

[AWS] Link Lambda and S3 with boto3
Connect to s3 with AWS Lambda Python
[AWS] Do SSI-like things with S3 / Lambda
S3 uploader with boto
Dynamic HTML pages made with AWS Lambda and Python
Manipulate S3 objects with Boto3 (high-level API and low-level API)
AWS Lambda with PyTorch [Lambda import]
Make ordinary tweets fleet-like with AWS Lambda and Python
Output CloudWatch Logs to S3 with AWS Lambda (Pythyon ver)
Site monitoring and alert notification with AWS Lambda + Python + Slack
[AWS] Create API with API Gateway + Lambda
Using Lambda with AWS Amplify with Go
Easy AWS S3 testing with MinIO
Notify HipChat with AWS Lambda (Python)
Use boto3 to mess with S3
Generate S3 signed URL with boto
[AWS] Search and acquire necessary data from S3 files with S3 Select
Prepare an environment to use OpenCV and Pillow with AWS Lambda
Install pip in Serverless Framework and AWS Lambda with Python environment
[AWS] Using ini files with Lambda [Python]
Try server-side encryption on S3 with boto3
S3 server-side encryption SSE with Python boto3
Touch AWS with Serverless Framework and Python
Python + Selenium + Headless Chromium with aws lambda
I just did FizzBuzz with AWS Lambda
[AWS] Create a Python Lambda environment with CodeStar and do Hello World
Easy server monitoring with AWS Lambda (Python) and result notification in Slack
Upload what you got in request to S3 with AWS Lambda Python
[AWS lambda] Deploy including various libraries with lambda (generate a zip with a password and upload it to s3) @ Python
[AWS SAM] Create API with DynamoDB + Lambda + API Gateway
MessagePack-Try to link Java and Python with RPC
Differences between queryStringParameters and multiValueQueryStringParameters in AWS Lambda
Regular serverless scraping with AWS lambda + scrapy Part 1.8
View images on S3 with API Gateway + Lambda
Try using S3 object upload and download with AWS SDK for Go v2
Serverless scraping using selenium with [AWS Lambda] -Part 1-
Get and parse S3 Logging logs with Lambda event nortification and plunge into BigQuery
LINE BOT with Python + AWS Lambda + API Gateway
Serverless application with AWS SAM! (APIGATEWAY + Lambda (Python))
Easy X-Ray with Lambda Layer and CloudFormation / sam-cli
Manage AWS nicely with the Python library Boto
[AWS; Introduction to Lambda] 2nd; Extract sentences from json file and save S3 ♬
[AWS] Try tracing API Gateway + Lambda with X-Ray
Export RDS snapshot to S3 with Lambda (Python)
I tried connecting AWS Lambda with other services
Describe ec2 with boto3 and retrieve the value
Infrastructure construction automation with CloudFromation + troposphere + AWS Lambda
Amazon API Gateway and AWS Lambda Python version
Develop, run, and deploy AWS Lambda remotely using lambda-uploader
[AWS] Play with Step Functions (SAM + Lambda) Part.3 (Branch)
Create Amazon Linux with AWS EC2 and log in
Deploy Python3 function with Serverless Framework on AWS Lambda
Issue S3 time-limited URL with boto3 (with file existence confirmation)
Create a Layer for AWS Lambda Python with Docker
[AWS] Play with Step Functions (SAM + Lambda) Part.1 (Basic)
I want to AWS Lambda with Python on Mac!
Manage your Amazon CloudWatch loggroup retention with AWS Lambda
Build AWS EC2 and RDS with Terraform Terraform 3 minutes cooking
[Memo] Load csv of s3 into pandas with boto3
[AWS] Play with Step Functions (SAM + Lambda) Part.2 (Parameter)
Launch Lambda on Boto3