I want to set a life cycle in the task definition of ECS

Introduction

When writing batch processing etc. using the task schedule function of ECS, if the task is updated many times using ʻecs-cli, the revision` of the task definition will increase.

It is troublesome to delete the revision (unregister) by clicking from the console screen of aws, and there is also a method to delete it using deregister-task-definition of ʻawscli`, but it is also troublesome to execute it from the command line. ..

How to delete with awscli https://docs.aws.amazon.com/cli/latest/reference/ecs/deregister-task-definition.html

So I decided to run Lambda to unregister (unregister) the revision.

What was used

--For deploying ServerlessFrameWork lambda --serverless-python-requirements For deploying external Python packages --Pipenv package management

Get a list of ECS task definitions

Use ecs.list_task_definitions in boto3 to get all the task definitions. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.list_task_definitions

I set sort ='DESC' as an argument, but I set it to 'DESC' because it is necessary when using ʻitertools.groupby` later.


import json
import boto3
import logging

logger = logging.Logger(__name__)
logger.setLevel(logging.DEBUG)

def get_task_definitions():
    """Get ECS task definition

    Returns:
        list --Task definition arn list
    """

    client = boto3.client('ecs')
    try:
        res = client.list_task_definitions(
            status='ACTIVE',
            sort='DESC')
        logger.info(json.dumps(res))
    except Exception as e:
        logger.info(e)
    else:
        logger.info('Successful task definition acquisition')

    list_task_definitions = res['taskDefinitionArns']
    return list_task_definitions

You can get it like this

[
    'arn:aws:ecs:ap-northeast-1:XXXXXXXXXXXXX:task-definition/task_a:3',
    'arn:aws:ecs:ap-northeast-1:XXXXXXXXXXXXX:task-definition/task_a:2',
    'arn:aws:ecs:ap-northeast-1:XXXXXXXXXXXXX:task-definition/task_a:1',
    'arn:aws:ecs:ap-northeast-1:XXXXXXXXXXXXX:task-definition/task_b:5',
    'arn:aws:ecs:ap-northeast-1:XXXXXXXXXXXXX:task-definition/task_b:4',
    'arn:aws:ecs:ap-northeast-1:XXXXXXXXXXXXX:task-definition/task_b:3',
    'arn:aws:ecs:ap-northeast-1:XXXXXXXXXXXXX:task-definition/task_b:2',
    'arn:aws:ecs:ap-northeast-1:XXXXXXXXXXXXX:task-definition/task_b:1',
    'arn:aws:ecs:ap-northeast-1:XXXXXXXXXXXXX:task-definition/task_c:5',
    'arn:aws:ecs:ap-northeast-1:XXXXXXXXXXXXX:task-definition/task_c:4',
    'arn:aws:ecs:ap-northeast-1:XXXXXXXXXXXXX:task-definition/task_c:3',
    'arn:aws:ecs:ap-northeast-1:XXXXXXXXXXXXX:task-definition/task_c:2',
    'arn:aws:ecs:ap-northeast-1:XXXXXXXXXXXXX:task-definition/task_c:1'
]

Format task definition data

Now that we have the task definition, we will group it by the same task name and format the data so that we can see how many revisions the task has. I got the task definition, but to be exact, it is arn of the task definition, so I format it only to the task name.


#The shape you want to shape
{
    'task_a': ['3', '2', '1'],
    'task_b': ['5', '4', '3', '2', '1'],
    'task_c': ['5', '4', '3', '2', '1']
}

ʻUse itertools.groupby to group them. I set sort in ʻecs.list_task_definitions because I need to sort in advance to use ʻitertools.groupby`.


a = [('a', '3'), ('a', '2'), ('a', '1'), ('b', '4'), ('b', '3'),
     ('b', '2'), ('b', '1'), ('c', '2'), ('c', '1')]

for key, group in groupby(a, lambda x: x[0]): 
    for team in group: 
        print(team)
    print('')

It will be grouped like this, so I will format it into a dictionary.

('a', '3')
('a', '2')
('a', '1')

('b', '4')
('b', '3')
('b', '2')
('b', '1')

('c', '2')
('c', '1')
from itertools import groupby

def groupby_task_definitions(list_task_definitions):
    """Put together a list of arns in the task definition

    Arguments:
        list_task_definitions {list} --Task definition arn list

    Returns:
        dict --Task name and revision list

    Example:
        {
            task_a: ['4', '3', '2', '1'],
            task_b: ['2', 1]
        }
    """
    group_list = []
    for task_definition in list_task_definitions:
        #Cut only the task name and revision
        task_definition = task_definition.rsplit('/', 1)[1]
        task_name, revision = task_definition.split(':')

        #Add to list as tuple
        group_list.append((task_name, revision))

    result_dict = {}
    for key, group in groupby(group_list, lambda x: x[0]):
        revision_list = []
        for _, v in group:
            revision_list.append(v)
        result_dict[key] = revision_list

    return result_dict

Delete ECS task (unregister)

Now that we have created a dictionary of task definitions, we will delete the ECS tasks. Turn the dictionary with for and delete it withpop ()until the length of the obtained list of revisions reaches the specified number.

ʻEcs.deregister_task_definition` can be executed easily by giving the name of the task definition to the argument.

def deregister_task_definition(groupby_task_definitions_dict, leave):
    """Delete the task definition, leaving up to the two latest revisions

    Arguments:
        groupby_task_definitions_dict {dict} --A dict that stores a list of task names and revisions
    """
    client = boto3.client('ecs')

    for name, revision_list in groupby_task_definitions_dict.items():
        logger.info(name)
        try:
            while len(revision_list) > 2:
                revision = revision_list.pop()
                client.deregister_task_definition(
                    taskDefinition='{}:{}'.format(name, revision)
                )
        except Exception as e:
            logger.error(e)
        else:
            logger.info('{} : OK!'.format(name))

Make it available on Lambda

We haven't set any events at this stage, but you can use lambda invoke or CloudWatch Events on a regular basis.

serverless.yml We will create Lambda using Serverless Framework, so expose serverless.yml.

serverless.yml


service: ecs-task

provider:
  name: aws
  runtime: python3.7
  region: ap-northeast-1
  logRetentionInDays: 30

  iamRoleStatements:
    - Effect: Allow
      Action:
        - ecs:*
      Resource: "*"

functions:
  definition-lifecycle :
    handler: handler.main

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    usePipenv: true 

handler The handler is just executing a function that does what it has done so far.

def main(event, context):
    list_task_definitions = get_task_definitions()
    groupby_task_definitions_dict = groupby_task_definitions(
        list_task_definitions)
    logger.info(groupby_task_definitions_dict)
    deregister_task_definition(groupby_task_definitions_dict, leave)

Whole code

handler.py



import boto3
import json
from itertools import groupby
import logging

logger = logging.Logger(__name__)
logger.setLevel(logging.DEBUG)


def get_task_definitions():
    """Get ECS task definition

    Returns:
        list --Task definition arn list
    """

    client = boto3.client('ecs')
    try:
        res = client.list_task_definitions(
            status='ACTIVE',
            sort='DESC')
        logger.info(json.dumps(res))
    except Exception as e:
        logger.info(e)
    else:
        logger.info('Successful task definition acquisition')

    list_task_definitions = res['taskDefinitionArns']
    return list_task_definitions


def groupby_task_definitions(list_task_definitions):
    """Put together a list of arns in the task definition

    Arguments:
        list_task_definitions {list} --Task definition arn list

    Returns:
        dict --Task name and revision list

    Example:
        {
            task_a: ['4', '3', '2', '1'],
            task_b: ['2', 1]
        }
    """
    group_list = []
    for task_definition in list_task_definitions:
        #Cut only the task name and revision
        task_definition = task_definition.rsplit('/', 1)[1]
        task_name, revision = task_definition.split(':')

        group_list.append((task_name, revision))

    result_dict = {}
    for key, group in groupby(group_list, lambda x: x[0]):
        revision_list = []
        for _, v in list(group):
            revision_list.append(v)
        result_dict[key] = revision_list

    return result_dict


def deregister_task_definition(groupby_task_definitions_dict):
    """Delete the task definition, leaving up to the two latest revisions

    Arguments:
        groupby_task_definitions_dict {dict} --A dict that stores a list of task names and revisions
    """
    client = boto3.client('ecs')

    for name, revision_list in groupby_task_definitions_dict.items():
        logger.info(name)
        try:
            while len(revision_list) > 2:
                revision = revision_list.pop()
                client.deregister_task_definition(
                    taskDefinition='{}:{}'.format(name, revision)
                )
        except Exception as e:
            logger.error(e)
        else:
            logger.info('{} : OK!'.format(name))


def main(event, context):
    list_task_definitions = get_task_definitions()
    groupby_task_definitions_dict = groupby_task_definitions(
        list_task_definitions)
    logger.info(groupby_task_definitions_dict)
    deregister_task_definition(groupby_task_definitions_dict)

AWS somehow

You can delete (unregister) a task definition only for tasks that are ʻActive, and then go into the ʻInactive state. In that state, it cannot be completely erased at this point, so we are waiting for AWS's response.

At this time, the INACTIVE task definition remains discoverable in your account indefinitely. However, this behavior is subject to change in the future, so make sure that you do not rely on the ACL task definition that is retained across the life cycle of the associated task and service.

image.png

https://docs.aws.amazon.com/ja_jp/AmazonECS/latest/developerguide/deregister-task-definition.html

Recommended Posts

I want to set a life cycle in the task definition of ECS
I want to sort a list in the order of other lists
I want to see a list of WebDAV files in the Requests module
I want to print in a comprehension
[Introduction to StyleGAN] I played with "The Life of a Man" ♬
I want to know the population of each country in the world.
I want to batch convert the result of "string" .split () in Python
I want to color a part of an Excel string in Python
I want to leave an arbitrary command in the command history of Shell
I made a program to check the size of a file in Python
I tried to display the altitude value of DTM in a graph
Python: I want to measure the processing time of a function neatly
I want to customize the appearance of zabbix
I want to display the progress in Python!
If you want to switch the execution user in the middle of a Fabric task, settings context manager
I want to use complicated four arithmetic operations in the IF statement of the Django template! → Use a custom template
Set the number of elements in a NumPy one-dimensional array to a power of 2 (0 padded)
The story of IPv6 address that I want to keep at a minimum
[Django memo] I want to set the login user information in the form in advance
I made an appdo command to execute a command in the context of the app
I want to add silence to the beginning of a wav file for 1 second
I want to store the result of% time, %% time, etc. in an object (variable)
I want to embed a variable in a Python string
I want to easily implement a timeout in python
I want to transition with a button in flask
I want to grep the execution result of strace
I want to write in Python! (2) Let's write a test
I want to randomly sample a file in Python
I want to fully understand the basics of Bokeh
I want to work with a robot in python.
I want to install a package of Php Redis
I want to write in Python! (3) Utilize the mock
I want to use the R dataset in python
I want to increase the security of ssh connections
The story of Linux that I want to teach myself half a year ago
I want to set up a mock server for python-flask in seconds using swagger-codegen.
I want to save a file with "Do not compress images in file" set in OpenPyXL
I want to take a screenshot of the site on Docker using any font
I want to use a network defined by myself in PPO2 of Stable Baselines
Find a guideline for the number of processes / threads to set in the application server
[Python] I want to get a common set between numpy
I want to use only the normalization process of SudachiPy
NikuGan ~ I want to see a lot of delicious meat! !!
I want to get the operation information of yahoo route
I made a function to check the model of DCGAN
I want to judge the authenticity of the elements of numpy array
I want to know the features of Python and pip
I want to make the Dictionary type in the List unique
Keras I want to get the output of any layer !!
I want to align the significant figures in the Numpy array
I want to know the legend of the IT technology world
I want to make input () a nice complement in python
I want to create a Dockerfile for the time being.
I didn't want to write the AWS key in the program
Make a note of what you want to do in the future with Raspberry Pi
[Python] I want to make a 3D scatter plot of the epicenter with Cartopy + Matplotlib!
I want to see the graph in 3D! I can make such a dream come true.
I want to find the intersection of a Bezier curve and a straight line (Bezier Clipping method)
I tried to create a Python script to get the value of a cell in Microsoft Excel
I want to output a beautifully customized heat map of the correlation matrix. matplotlib edition
I wrote a doctest in "I tried to simulate the probability of a bingo game with Python"