How to run a job with docker login in AWS batch

background

"Isn't it okay to have an image of such an ECR?", But I thought there was someone like me who had already designed the entire repository to manage the repository on Docker Hub. Or maybe there are people who manage the docker repository on their own.

Prerequisites

-Docker image is registered in the place where authentication is required like Docker Hub -An instance of Amazon Linux is running on EC2 ・ Jobs can be started using AWS batch

procedure

Meet ECS requirements with an instance of EC2

The AMI that can be launched from batch must meet the following requirements (Manual here).

So, access the running EC2 instance with ssh and work.

$ ssh -i ec2.key ec2-user@[EC2 IP address]
#Install the ECS container agent.
$ sudo amazon-linux-extras install -y ecs; sudo systemctl enable --now ecs
#Install docker.
$ yum install -y docker

You have met your requirements.

Create a Docker Hub authenticated AMI

Next, authenticate Docker Hub so that you can use that authentication information when you start it with batch.

#Log in to Docker Hub
$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don’t have a Docker ID, head over to https://hub.docker.com to create one.
Username: xxxxxxxxx
Password:
Login Succeeded

When you log in, the following login information will be generated in "~/.docker/config.json". The following is the case when you log in to Docker Hub. (Description method when using other repositories or multiple repositories)

{
  "auths": {
    "https://index.docker.io/v1/": {
      "auth": "zq212MzEXAMPLE7o6T25Dk0i"
    }
  }
}

Describe this in the configuration file that batch reads when starting EC2 as shown below.

$ sudo vi /etc/ecs/ecs.config
#Fill in the following
ECS_ENGINE_AUTH_TYPE=dockercfg
ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/":{"auth":"zq212MzEXAMPLE7o6T25Dk0i","email":"[email protected]"}}

If you can do it so far, check the EC2 you set from the console screen of EC2 and select "Action"-> "Image and template"-> "Create image", specify the image name and create an AMI.

Job registration by specifying AMI

Once you have registered your AMI, you can use it to launch jobs. First, go to "Computing Environment"-> "Create" on the console screen of AWS Batch. Then, open "Additional settings: Template, launch user-specified AMI" at the bottom of the screen, and check "Enable" for "User-specified AMI ID". Then, a text box for entering the AMI ID will be displayed as shown in the image below, so paste the ID of the AMI you registered earlier.

スクリーンショット 2020-12-14 12.47.39.png

After setting other required items, click "Create Computing Environment" to register the computing environment. Next, go to "Job Queue"-> "Create" and register the job queue using the computing environment you registered earlier.

スクリーンショット 2020-12-14 12.53.18.png

Enter the required items and click "Create" to complete the job queue registration. Finally, specify the image of the private repository you want to use in the job definition.

スクリーンショット 2020-12-14 12.56.23.png

Enter the required items and click "Create" to complete the job definition registration.

After that, if you register a new job using the job queue and job definition, you can execute the job definition command in the docker container that uses the image of the private repository.

Reference URL

https://docs.aws.amazon.com/batch/latest/userguide/compute_resource_AMIs.html#batch-ami-spec https://docs.aws.amazon.com/AmazonECS/latest/developerguide/private-auth-container-instances.html#docker-auth-formats https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html

Recommended Posts

How to run a job with docker login in AWS batch
How to start a Docker container with a volume mounted in a batch file
Submit a job to AWS Batch with Java (Eclipse)
How to run Blazor (C #) with Docker
How to run a djUnit task in Ant
How to migrate a web application created in a local docker environment to AWS
How to delete untagged images in bulk with Docker
How to use In-Memory Job repository in Spring Batch
Mapping to a class with a value object in How to MyBatis
How to set up a proxy with authentication in Feign
Write DiscordBot to Spreadsheets Write in Ruby and run with Docker
How to run JUnit in Eclipse
How to run Ant in Gradle
How to run JavaFX on Docker
How to start Camunda with Docker
How to make a jar file with no dependencies in Maven
How to implement a job that uses Java API in JobScheduler
How to rename a model with foreign key constraints in Rails
Run (provisionally) a Docker image with ShellCommandActivity on AWS Data Pipeline
How to share files with Docker Toolbox
[Rails] How to use rails console with docker
How to insert a video in Rails
How to use Docker in VSCode DevContainer
How to publish a library in jCenter
Understand in 5 minutes !! How to use Docker
How to build Rails 6 environment with Docker
How to quickly create a reverse proxy that supports HTTPS with Docker
[Personal memo] How to interact with a random number generator in Java
[Rails] How to log in with a name by adding a devise name column
How to execute with commands of normal development language in Docker development environment
I tried to build a Firebase application development environment with Docker in 2020
How to build a Ruby on Rails development environment with Docker (Rails 6.x)
[AWS CLI] Switch roles to get a Docker image in another account
How to build a Ruby on Rails development environment with Docker (Rails 5.x)
How to check the logs in the Docker container
How to deploy a container on AWS Lambda
How to get a heapdump from a Docker container
How to use Java framework with AWS Lambda! ??
I made a Docker container to run Maven
How to add a classpath in Spring Boot
How to update pre-built files in docker container
How to create a theme in Liferay 7 / DXP
How to give your image to someone with docker
How to implement a like feature in Rails
How to easily create a pull-down in Rails
How to use docker compose with NVIDIA Jetson
How to use nginx-ingress-controller with Docker for Mac
[Rails] How to build an environment with Docker
How to make a follow function in Rails
How to redirect after user login with Spring-security
How to automatically generate a constructor in Eclipse
How to get the ID of a user authenticated with Firebase in Swift
Until you run a Java program with the AWS SDK local to Windows
How to store data simultaneously in a model associated with a nested form (Rails 6.0.0)
How to embed JavaScript variables in HTML with Thymeleaf
How to implement UICollectionView in Swift with code only
How to clear all data in a particular table
How to sort in ascending / descending order with SQLite
How to make batch processing with Rails + Heroku configuration
How to make a factory with a model with polymorphic association
How to create a Java environment in just 3 seconds