Export RDS snapshot to S3 with Lambda (Python)

Introduction

When referencing RDS data

-There is no date and time dump job etc. -No read replica ・ No questions asked and approval is required to log in to a commercial environment.

For some reason, it was sometimes difficult to look directly into the contents of RDS. At that time, it was said that it was possible to export the snapshot to S3 to Amazon RDS and refer to it with Athena, so I actually tried S3 export of the RDS snapshot using Lambda.

Overall picture

Export RDS snapshot to S3 with Lambda. Since the exported data is encrypted by KMS, it is necessary to prepare the KMS key as well.

Premise

-RDS and snapshots have been created. -The export destination S3 bucket has been created.

procedure

1. Create policies and roles to use when exporting RDS snapshots to S3

1-1. Policy creation

Create a policy to access S3 from the snapshot export task with the following json. (Resource your-s3-bucket specifies the S3 bucket to export to)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject*",
                "s3:GetObject*",
                "s3:DeleteObject*"
            ],
            "Resource": [
                "arn:aws:s3:::your-s3-bucket",
                "arn:aws:s3:::your-s3-bucket/*"
            ]
        }
    ]
}

1-2. Role creation

Create a role and attach the policy created above. From Edit Trust Relationship, edit the access control policy document as follows. (Service becomes export.rds.amazonaws.com)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "export.rds.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

2. Creating a KMS key

Create a key in KMS to use to encrypt the export data.

1. Select Customer managed key in KMS from the management console.
2. Display the creation wizard with Create key.
3. Step 1/5 Key type: Target
4. Step 2/5 Alias: Enter the name of any key
5. Step 3/5 Key Administrator: Select the key and the user who will be the administrator
6. Step 4/5 Define key usage permissions: Not specified here
7. Step 5/5 Complete

3. Create Lambda

3-1. Role, policy creation

Create a role for AWSLambdaBasicExecutionRole that can be created at the same time you create a Lambda function. Create the following policy required for snapshot export separately and attach it to the above role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:PassRole",
                "rds:StartExportTask"
            ],
            "Resource": "*"
        }
    ]
}

3-2. Lambda code creation

pyhton：lambda_handler.py

import json
import boto3
from datetime import datetime

SOURCE_ARN="YOUR_SOURCE_ARN"
S3_BUCKET_NAME="YOUR_S3_BUCKET_NAME"
IAM_ROLE_ARN="YOUR_IAM_ROLE_ARN"
KMS_KEY_ID="YOUR_KMS_KEY_ID"

client = boto3.client('rds')

def lambda_handler(event, context):
  
    export_task_identifier="mysnapshot" + datetime.now().strftime("%Y%m%d%H%M%S")
    
    response = client.start_export_task(
        ExportTaskIdentifier=export_task_identifier,
        SourceArn=SOURCE_ARN,
        S3BucketName=S3_BUCKET_NAME,
        IamRoleArn=IAM_ROLE_ARN,
        KmsKeyId=KMS_KEY_ID,
        )

-Set the following variables in the code according to your environment.

-Since the following restrictions apply to ExportTaskIdentifier, the date and time are added this time.

Export identifier Enter a name to identify the export. This name must be unique across all snapshot exports that your AWS account owns in your current AWS Region.

4. Add Lambda role to key user of KMS key

Add the role used in Lambda in step 3 to the key user of the key created in step 2.

If you don't do this, you'll get the error An error occurred (KMSKeyNotAccessibleFault) when you run Lambda.

After this work, run Lambda to export the RDS snapshot to S3.

in conclusion

You can do the same with the management console if you just want to export the snapshot to S3, but I wanted to make the data source a system snapshot and update the data with the date and time, so I tried running it with Lambda.

It would be nice if RDS could be checked on the management console if it was a small reference like DynamoDB, but I felt that the method of referring to this snapshot is also a safe and easy method.

References

This article was written with reference to the following information.

-Export DB Snapshot Data to Amazon S3