How to run a job with docker login in AWS batch

background

"Isn't it okay to have an image of such an ECR?", But I thought there was someone like me who had already designed the entire repository to manage the repository on Docker Hub. Or maybe there are people who manage the docker repository on their own.

Prerequisites

-Docker image is registered in the place where authentication is required like Docker Hub -An instance of Amazon Linux is running on EC2 ・ Jobs can be started using AWS batch

procedure

Meet ECS requirements with an instance of EC2

The AMI that can be launched from batch must meet the following requirements (Manual here).

• The ECS container agent is running
• Logs can be output to CloudWatch
• You can use docker

So, access the running EC2 instance with ssh and work.

$ ssh -i ec2.key ec2-user@[EC2 IP address]
#Install the ECS container agent.
$ sudo amazon-linux-extras install -y ecs; sudo systemctl enable --now ecs
#Install docker.
$ yum install -y docker

You have met your requirements.

Create a Docker Hub authenticated AMI

Next, authenticate Docker Hub so that you can use that authentication information when you start it with batch.

#Log in to Docker Hub
$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don’t have a Docker ID, head over to https://hub.docker.com to create one.
Username: xxxxxxxxx
Password:
Login Succeeded

When you log in, the following login information will be generated in "~/.docker/config.json". The following is the case when you log in to Docker Hub. (Description method when using other repositories or multiple repositories)

{
  "auths": {
    "https://index.docker.io/v1/": {
      "auth": "zq212MzEXAMPLE7o6T25Dk0i"
    }
  }
}

Describe this in the configuration file that batch reads when starting EC2 as shown below.

$ sudo vi /etc/ecs/ecs.config
#Fill in the following
ECS_ENGINE_AUTH_TYPE=dockercfg
ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/":{"auth":"zq212MzEXAMPLE7o6T25Dk0i","email":"[email protected]"}}

If you can do it so far, check the EC2 you set from the console screen of EC2 and select "Action"-> "Image and template"-> "Create image", specify the image name and create an AMI.

Job registration by specifying AMI

Once you have registered your AMI, you can use it to launch jobs. First, go to "Computing Environment"-> "Create" on the console screen of AWS Batch. Then, open "Additional settings: Template, launch user-specified AMI" at the bottom of the screen, and check "Enable" for "User-specified AMI ID". Then, a text box for entering the AMI ID will be displayed as shown in the image below, so paste the ID of the AMI you registered earlier.

After setting other required items, click "Create Computing Environment" to register the computing environment. Next, go to "Job Queue"-> "Create" and register the job queue using the computing environment you registered earlier.

Enter the required items and click "Create" to complete the job queue registration. Finally, specify the image of the private repository you want to use in the job definition.

Enter the required items and click "Create" to complete the job definition registration.

After that, if you register a new job using the job queue and job definition, you can execute the job definition command in the docker container that uses the image of the private repository.

Reference URL

https://docs.aws.amazon.com/batch/latest/userguide/compute_resource_AMIs.html#batch-ami-spec https://docs.aws.amazon.com/AmazonECS/latest/developerguide/private-auth-container-instances.html#docker-auth-formats https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html