Manage your Amazon CloudWatch loggroup retention with AWS Lambda

Hello, this is risako of stremapack. After a long rainy season, summer has arrived: sunny: You need to be more careful not to get heat stroke with a mask while being careful of the corona: mask: I will spend this summer taking care of my physical condition.

What I want to do this time

The theme this time is "Let's manage Amazon CloudWatch log group retention period with AWS Lambda"! When using Amazon ECS or AWS Lambda (hereinafter Lambda), logs continue to accumulate in Amazon CloudWatch (hereinafter CoudWatch). What's more, the CloudWatch LogGroup retention period is set to "Never expire" by default, so logs will remain forever and the amount will increase. This time, in order to reduce the amount of money as much as possible, I would like to introduce a mechanism in Lambda that automatically changes the retention period of the newly created log group to "1 month"!

This item

let's try it

IAM Role settings

The Role name should be ** cloudwatchtest-lambda-role **.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

IAM Policy settings

The Policy name is ** cloudwatchtest-lambda-policy **. You have been granted permission to operate CloudWatch.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:PutRetentionPolicy",
                "logs:CreateLogGroup",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]

Lambda settings

The Lambda function was created below. ** Function name: change_Retention_for_cloudwatch-logs Runtime: Python 3.8 **

Function code I want to change the retention period of the newly created log group to ** 1 month (30 days) **, so set retentionInDays = 30. Here you specify the desired retention period. Example: 1.3.5.7.60

import boto3

logs = boto3.client('logs')

def lambda_handler(event, context):

    loggroupname = event['detail']['requestParameters']['logGroupName']

    try:
        response = logs.put_retention_policy(
            logGroupName = loggroupname,
            retentionInDays = 30
        )

    except Exception as e:
        print(e)

Event Bridge settings

This time, I want to move the Lambda created above with the trigger that CloudWatch Log Group is newly created, so let's have it detected by the rules of Event Bridge. Now let's create a rule.

The pattern definition is set below. ** Event pattern Event Match Pattern: Custom Pattern **

Put the code to be defined in the square on the right side. スクリーンショット 2020-08-06 17.52.02.png

For the target, specify the Lambda function you created earlier. スクリーンショット 2020-08-06 18.03.01.png

This completes the settings! Let's move

If the retention period of the newly created Cloudwatch LogGroup is one month, the operation check is OK! Create a new one from "Create Log Group" in the CloudWatch console.

** Immediately after creation ** スクリーンショット 2020-07-30 17.46.24.png

** After a few minutes ** スクリーンショット 2020-07-30 17.46.45.png

The retention period is not reflected immediately after creating a new one, but you can confirm that the retention period has been set correctly. This seems to prevent the log from accumulating and unknowingly increasing the price: ok_hand:

reference

・ What is Amazon Event Bridge? https://docs.aws.amazon.com/ja_jp/eventbridge/latest/userguide/what-is-amazon-eventbridge.html ・ What is Amazon CloudWatch Logs? https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html · How do I find out why I was charged for CloudWatch and then reduce future charges? https://aws.amazon.com/jp/premiumsupport/knowledge-center/cloudwatch-understand-and-reduce-charges/?nc1=h_ls

Recommended Posts

Manage your Amazon CloudWatch loggroup retention with AWS Lambda
Manage your data with AWS RDS
AWS Lambda with PyTorch [Lambda import]
Output CloudWatch Logs to S3 with AWS Lambda (Pythyon ver)
[AWS] Create API with API Gateway + Lambda
Periodically start / stop Sakura's cloud server with AWS Lambda + CloudWatch Events
Using Lambda with AWS Amplify with Go
Notify HipChat with AWS Lambda (Python)
[AWS] Using ini files with Lambda [Python]
[AWS] Link Lambda and S3 with boto3
Connect to s3 with AWS Lambda Python
[AWS] Do SSI-like things with S3 / Lambda
Python + Selenium + Headless Chromium with aws lambda
I just did FizzBuzz with AWS Lambda
Try implementing a Cisco Spark bot with AWS Lambda + Amazon API Gateway (Python)
[AWS SAM] Create API with DynamoDB + Lambda + API Gateway
Regular serverless scraping with AWS lambda + scrapy Part 1.8
Declaratively manage your environment with Nix and home-manager
LINE BOT with Python + AWS Lambda + API Gateway
Serverless application with AWS SAM! (APIGATEWAY + Lambda (Python))
Manage AWS nicely with the Python library Boto
[AWS] Try tracing API Gateway + Lambda with X-Ray
I tried connecting AWS Lambda with other services
Infrastructure construction automation with CloudFromation + troposphere + AWS Lambda
Amazon API Gateway and AWS Lambda Python version