Infrastructure construction automation with CloudFromation + troposphere + AWS Lambda

I created a template creation tool for CloudFormation using troposphere. Also, by running the created tool on Lambda, I tried to create a template just by placing the parameter sheet on S3.

JSON format is hard to write.

Recently, I had the opportunity to touch CloudFormation in my business. I think the bottleneck in using CloudFormation is the JSON format file.

python


{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "hogehoge",
    "Resources": {
        "test1": {
            "Properties": {
                "CidrBlock": "10.1.0.0/16",
                "EnableDnsHostnames": "true",
                "EnableDnsSupport": "true",
                "Tags": [
                    {
                        "Key": "Name",
                        "Value": "test1"
                    }
                ]
            },
            "Type": "AWS::EC2::VPC"
        },
        "test2": {
            "Properties": {
                "CidrBlock": "10.2.0.0/16",
                "EnableDnsHostnames": "true",
                "EnableDnsSupport": "true",
                "Tags": [
                    {
                        "Key": "Name",
                        "Value": "test2"
                    }
                ]
            },
            "Type": "AWS::EC2::VPC"
        }
    }
}

It's like this just to make two VPCs, so I think it's quite a craftsmanship to write this by hand.

What is troposphere?

troposphere is a tool to create such a JSON format file by writing it in Python. https://github.com/cloudtools/troposphere If the above code is also written using troposphere, it can be written in an easy-to-read manner as follows.

python


import json
from troposphere import Tags,Template
from troposphere.ec2 import VPC
t = Template()
t.add_version("2010-09-09")
t.add_description("hogehoge")
t.add_resource(VPC(
              "test1",
              EnableDnsSupport="true",
              CidrBlock="10.1.0.0/16",
              EnableDnsHostnames="true",
              Tags=Tags(
                    Name="test1"
                    )
              )) 
t.add_resource(VPC(
              "test1",
              EnableDnsSupport="true",
              CidrBlock="10.2.0.0/16",
              EnableDnsHostnames="true",
              Tags=Tags(
                    Name="test2"
                    )
              ))
json_template = t.to_json()
print(json_template)                

The basic usage is to describe the version name and description with add_version and add_description, and then add more resources to be created with add_resource. It's not dramatically easier to see, but I think it's a little better than writing it in JSON format. Also, not only is it easier to see, but the point is that you can use Python for and if statements. It is possible to easily describe a large amount of resources by describing by passing multiple CIDR blocks and Names as an array.

python


import json
from troposphere import Tags,Template
from troposphere.ec2 import VPC
t = Template()
t.add_version("2010-09-09")
t.add_description("hogehoge")
VPC_CidrBlockList = ["10.1.0.0/16","10.2.0.0/16"]
VPC_TagsNameList = ["test1","test2"]
for (Address,Tag_Name) in zip(VPC_CidrBlockList,VPC_TagNameList):
t.add_resource(VPC(
              Tag_Name,
              EnableDnsSupport="true",
              CidrBlock=Address,
              EnableDnsHostnames="true",
              Tags=Tags(
                    Name=Tag_Name
                    )
              )) 
json_template = t.to_json()
print(json_template)                

Bring the set value from the outside.

By the way, in the above state, setting values such as CIDR block and Name will be written in a script, but this is difficult to use. Let's read the parameters from the outside and store them in an array so that they can be used. In this case, there are two types of information required as parameters (CIDR block, Name), so create a parameter sheet as shown below and save it as a text file. 1st line 10.1.0.0/16, 10.2.0.0/16 2nd line test1, test2

After creating a text file, first open and read this text file.

python


f = open('test.txt')
test = f.read()
f.close

The data is now stored in test. However, in this state, the first line and the second line are not separated. So, use splitlines to store each row as an array.

python


test_list= test.splitlines()

This will create an array test_list with each row as an element. ["10.1.0.0/16,10.2.0.0/16","test1,test2"] Furthermore, here, split is used to create an array of each parameter.

python


VPC_CidrBlockList = test_list[0].split(',')
VPC_TagNameList = test_list[1].split(',')

Now you have created an array of parameters. If you incorporate these operations and modify the script above, it will be as follows.

python


import json
from troposphere import Tags,Template
from troposphere.ec2 import VPC

f = open('test.txt')
test = f.read()
f.close
test_list= test.splitlines()
VPC_CidrBlockList = test_list[0].split(',')
VPC_TagNameList = test_list[1].split(',')

t = Template()
t.add_version("2010-09-09")
t.add_description("hogehoge")
for (Address,Tag_Name) in zip(VPC_CidrBlockList,VPC_TagNameList):
t.add_resource(VPC(
              Tag_Name,
              EnableDnsSupport="true",
              CidrBlock=Address,
              EnableDnsHostnames="true",
              Tags=Tags(
                    Name=Tag_Name
                    )
              )) 
json_template = t.to_json()
print(json_template)                

You can now create multiple VPCs according to the parameters listed in the parameter sheet.

Try using AWS Lambda.

The basic part of using Lambda is the same, but the process of getting the parameter sheet from the S3 bucket and the process of uploading the created template to the S3 bucket are added.

python


# coding: utf-8

import json
import urllib
import boto3
from troposphere import Tags,Template
from troposphere.ec2 import VPC
from datetime import datetime

basename = datetime.now().strftime("%Y%m%d-%H%M%S")

print('Loading function')

s3 = boto3.resource('s3')

def lambda_handler(event, context):
        #print("Received event: " + json.dumps(event, indent=2))

        # Get the object from the event and show its content type
        bucket = event['Records'][0]['s3']['bucket']['name']
        key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key']).decode('utf8')
        print ("buket:" + bucket)
        print ("key:" + key)
        obj = s3.Object(bucket,key)
        response= obj.get()
        body = response['Body'].read()
        body_list= body.splitlines()
        # u'Creating a VPC'
        VPC_CidrBlockList = body_list[0].split(',')
        VPC_TagNameList= body_list[1].split(',')

        t = Template()
        t.add_version("2010-09-09")
        t.add_description("hogehoge")
        for (Address,Tag_Name) in zip(VPC_CidrBlockList,VPC_TagNameList):
            t.add_resource(VPC(
                  Tag_Name,
                  EnableDnsSupport="true",
                  CidrBlock=Address,
                  EnableDnsHostnames="true",
                  Tags=Tags(
                        Name=Tag_Name
                  )
            )) 
        json_template = t.to_json()
        bucket = s3.Bucket('template_test')
        obj = bucket.Object('json-template-' + basename + ' .txt')
        response = obj.put(
                       Body=json_template.encode('utf-8'),
                       ContentEncoding='utf-8',
                       ContentType='text/plane'
                    )
        print(json_template)                

You can also set it more flexibly by changing the items to be set as variables (such as "EnableDnsHostnames" in VPC). Also, if you describe other resources in this way, you can automatically create templates such as subnets and security groups as well as VPCs.

Recommended Posts

Infrastructure construction automation with CloudFromation + troposphere + AWS Lambda
Infrastructure construction automation with CloudFromation + troposphere + AWS Lambda
Environment construction: GCP + Docker
Install Docker on AWS
Flask site construction memorandum
AWS Lambda with PyTorch [Lambda import]
ruby environment construction with aws EC2
[AWS] Create API with API Gateway + Lambda
Using Lambda with AWS Amplify with Go
Notify HipChat with AWS Lambda (Python)
[AWS] Using ini files with Lambda [Python]
Connect to s3 with AWS Lambda Python
[AWS] Do SSI-like things with S3 / Lambda
Python + Selenium + Headless Chromium with aws lambda
I just did FizzBuzz with AWS Lambda
[AWS SAM] Create API with DynamoDB + Lambda + API Gateway
Regular serverless scraping with AWS lambda + scrapy Part 1.8
Serverless scraping using selenium with [AWS Lambda] -Part 1-
LINE BOT with Python + AWS Lambda + API Gateway
Serverless application with AWS SAM! (APIGATEWAY + Lambda (Python))
[AWS] Try tracing API Gateway + Lambda with X-Ray
I tried connecting AWS Lambda with other services
Dynamic HTML pages made with AWS Lambda and Python
[AWS] Play with Step Functions (SAM + Lambda) Part.3 (Branch)
Deploy Python3 function with Serverless Framework on AWS Lambda
Create a Layer for AWS Lambda Python with Docker
[AWS] Play with Step Functions (SAM + Lambda) Part.1 (Basic)
I want to AWS Lambda with Python on Mac!
Manage your Amazon CloudWatch loggroup retention with AWS Lambda
Make ordinary tweets fleet-like with AWS Lambda and Python
[AWS] Play with Step Functions (SAM + Lambda) Part.2 (Parameter)