[DOCKER] Lambda on Terraform Container

Introduction

This article is the 15th day of terraform Advent Calendar 2020.

Originally I tried to write about Terraform testing, but at re: Invent2020 it was Lambda container support. Among the announcements made at re: Invent 2020 this time, I think it was a relatively responsive announcement. I have to ride this wave, so I tried to containerize Terraform so that it can be executed on Lambda

Constitution

The configuration this time is as follows lambda_terraform.png

In development, Cloud9 is used, confirmation is done internally using a container, and in production, the image is uploaded to ECR and used from Lambda. API Gateway is unused because I didn't have time this time

Use Case

I think there are two use cases as below.

--Common execution platform --Monitoring

As a common execution platform, I think that there are people who often worry about the execution platform of Terraform, but I think that it is one idea to use Lambda, which is created this time, as a common execution platform. This time I just bring the code and hit it, but if you pull the code managed by CodeCommit or Github, I think that you can get the latest code and deploy it.

As for monitoring, I think that there is often a problem that the code and resources are separated, but if you use Lambda this time, you can detect if there is a separation. If you hit the API created this time, you can get the number of resources to add / change / delete, so if you use it with Prometheus or Datadog, you can get an alert when it is not 0.

Repository

The target repositories are as follows https://github.com/Yusuke-Shimizu/terraform_on_lambda If you are tired of reading articles, please clone and use it. Please modify the values ​​of Dockerfile and .envrc by yourself (Do not push with the access key and secret key written!)

Container image

If you want to put the container on Lambda, you need to build it with one of the following

-Use Base Image --Implemented Lambda Runtime API

The runtime API needs to create the necessary API, so it is flexible but time-consuming. Since the base image is easier just to put it in FROM and add it, I will use this this time

Selection of base image

This time we will run Terraform using Python Since Terraform itself is Go, I tried to do it with Go, but I gave up because I have little experience with Ikansen Go and there is little information on Lambda on Go. .. The base image was decided as follows with reference to Amazon ECR Public Gallery.

FROM public.ecr.aws/lambda/python:3.8
...

By the way, the public.ecr.aws of this FROM is also ECR Public announced at re: Invent2020. Simply put, it's the AWS version of Docker Hub.

Terraform installation

The above Python image can only run Python, so you need to be able to run Terraform Therefore, install the zip as shown below so that you can execute terraform.

ARG terraform_version="0.14.2"
ADD https://releases.hashicorp.com/terraform/${terraform_version}/terraform_${terraform_version}_linux_amd64.zip terraform_${terraform_version}_linux_amd64.zip
RUN yum -y install unzip wget
RUN unzip ./terraform_${terraform_version}_linux_amd64.zip -d /usr/local/bin/
RUN rm -f ./terraform_${terraform_version}_linux_amd64.zip
ENV TF_DATA_DIR /tmp

Execution directory

When I run Lambda, I run it in the/var/task directory, but basically I can't create or modify files in this directory Reference: https://stackoverflow.com/questions/45608923/aws-lambda-errormessage-errno-30-read-only-file-system-drive-python-quic

Therefore, the last ENV TF_DATA_DIR/tmp changes the data output destination of TF. Reference: https://www.terraform.io/docs/commands/environment-variables.html

The rest of the Dockerfile

All you have to do is copy the AWS key and necessary files and run the handler to complete the Dockerfile. Originally, the AWS key should use the Lambda role, but I want to perform local verification as well, so I put it in for the time being. Originally, I think that it is better to make it a type that injects environment variables from the outside as soon as it is converted to sts.

ENV AWS_ACCESS_KEY_ID [access key]
ENV AWS_SECRET_ACCESS_KEY [secret key]

# copy files
COPY app.py ${LAMBDA_TASK_ROOT}
COPY main.tf ${LAMBDA_TASK_ROOT}

CMD [ "app.handler" ]

Python For Python code, just create the handle function in app.py What I'm doing is copying the tf file to/tmp, initializing terraform and running the plan The reason for moving to/tmp is that a file is created in the current directory when $ terraform init is executed, but as explained above, files can only be created in/tmp, so move it. I'm trying to run it

def handler(event, context): 
    cmd('./', "cp ./main.tf /tmp/")
    cmd('/tmp/', "terraform init --upgrade")
    result = cmd('/tmp/', "terraform plan")
    last_result = extraction(result)
    return last_result

As for the result of plan, the number of add and change is taken as a regular expression below and returned as dict.

def extraction(plan):
    print(plan)
    change_state = {'add': 0, 'change': 0, 'destroy': 0}
    if "No changes" in plan:
        return change_state
    elif "Plan" in plan:
        line_extraction = re.findall("Plan.*", plan)
        result = "".join(line_extraction)
        
        change_state['add'] = int(re.findall('(\d)\sto\sadd', result)[0])
        change_state['change'] = int(re.findall('(\d)\sto\schange', result)[0])
        change_state['destroy'] = int(re.findall('(\d)\sto\sdestroy', result)[0])
        return change_state
    elif "Error" in plan:
        line_extraction = re.findall("Error.*", plan)
        result = "".join(line_extraction)
        return result
    else:
        result = "This is an unexpected error. Please check the log."
        return result

Terraform Terraform code made it easy to create an S3 bucket

provider "aws" {
  region  = "ap-northeast-1"
}

resource "aws_s3_bucket" "default" {
  bucket = "created-by-lambda"
  acl    = "private"
}

Run locally (Cloud9)

Start the container by executing as below

$ export REPOSITORY_NAME=[Repository name]
$ docker build -t ${REPOSITORY_NAME} .
$ docker run --rm -p 9000:8080 ${REPOSITORY_NAME}

Then, when you hit the local port 9000, S3 is created as shown below, so it returned with 1 in add.

$ curl -sd '{}' http://localhost:9000/2015-03-31/functions/function/invocations | jq .
{
  "add": 1,
  "change": 0,
  "destroy": 0
}

Up to ECR

Create an ECR repository with the name REPOSITORY_NAME set above After that, if you push the image below, the image will be latest up to ECR

$ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity | jq -r .Account)
$ export REGISTRY_URL=${AWS_ACCOUNT_ID}.dkr.ecr.ap-northeast-1.amazonaws.com
$ aws ecr get-login-password --region ap-northeast-1 | docker login --username AWS --password-stdin ${REGISTRY_URL}
$ docker tag ${REPOSITORY_NAME} ${REGISTRY_URL}/${REPOSITORY_NAME}
$ docker push ${REGISTRY_URL}/${REPOSITORY_NAME}

Run on Lambda

Creating Lambda

First, create an execution platform Lambda However, you can do it from the console. I think sam or CDK is better for automation (I personally recommend CDK)

From the console, select the container image from the function creation, decide the function name appropriately, and select the latest of the target repository from the "Browse image" button for the container image.

One thing to note is that the execution of Terraform this time takes time and consumes some memory, so please set as follows. Timeout: 1 minute Memory: 4GB

Run

Now that you're ready, let's finally hit the container Terraform from Lambda. This can also be lightly tested from the console, so let's try it here

Any test event is fine, so I tried running it using the default one Then, the plan result of Terraform is returned in json as below. スクリーンショット 2020-12-15 18.50.57.png If you use this, you can hit this API from the outside and monitor whether there is a difference with the current resource.

at the end

Now that Lambda can be run in a container, I put Terraform in and ran it In the future, I would like to link with API Gateway to make it an API, monitor Terraform, and create an application that executes not only plan but also apply.

Recommended Posts

Lambda on Terraform Container
How to deploy a container on AWS Lambda
Run React on a Docker container
Is Java on AWS Lambda slow?
Hello World on AWS Lambda + Java
Run GUI application on Docker container
Run PureScript on a Docker container
I tried deploying a Docker container on Lambda with Serverless Framework
Invalid lambda deserialization on Apache Wicket 8 Lambda
Run C binaries on AWS Lambda
Launch docker container on EC2 (personal memorandum)
Run NordVPN on Docker (Windows) Ubuntu container
Try running Word2vec model on AWS Lambda
Try building a GPU container on GCP.