[DOCKER] [Challenge CircleCI from 0] Understand AWS ECR / ECS
background
I aim to get a job at an in-house developed company from inexperienced. I decided to study CircleCI to create a good portfolio.
The current level of knowledge is that you can easily develop applications using Ruby on rails, version control using git, and deploy using heroku. I hope it will be of some help to those who are thinking of trying CircleCI in the future at the same level as their own memorandum.
final goals
Understand AWS ECR / ECS. Try deploying the application manually using ECR and ECS.
【Related article】 [Challenge CircleCI from 0] Learn the basics of CircleCI [Challenge CircleCI from 0] Build an automated test (Rails6.0 / mysql8.0 / Rspec) [Challenge CircleCI from 0] Understand AWS ECR / ECS [Challenge CircleCI from 0] Automatically deploy with CircleCI / AWS (ECR / ECS)
environment
ruby 2.6.6 rails 6.0 db: mysql 8.0 test: rspec
table of contents
- What is ECR
- What is ECS
- Actually deploy ECR / ECS on AWS
1. What is ECR
About ECR
ECR is an abbreviation of "Elastic Container Registry" and is a container image management system on AWS. If you've used Docker before, you probably know DockerHub, a cloud registry service, but in a nutshell, ECR is the AWS version of DokcerHub.
ECR is a very useful service when deploying with containers on AWS. Of course, you can also operate using Docker Hub, but when using AWS in a production environment, it is easier to manage using ECR. ECR is a service managed by Amazon, so its security is guaranteed.
There are no new concepts or terms for ECR, and if you've used Docker Hub, you'll understand it right away.
2. What is ECS?
About ECS
ECS is an abbreviation of "Elastic Container Service" and is a container execution and management system on AWS. In fact, the ECR mentioned above is also a kind of ECS service. ECS is a system that allows you to easily start, stop, and manage containers on AWS.
If you want to use a container to deploy your application to AWS production, ECS is a must-have service. However, there are some unique concepts and terms that are difficult to get started with. So first, let's understand the terms and concepts unique to ECS.
About ECS concept and terminology
cluster
An Amazon ECS cluster is a logical grouping of tasks or services. If you're using EC2 to perform tasks or services, your cluster is also a grouping of container instances. If you are using a capacity provider, the cluster is also a logical group of capacity providers.
In AWS Official, it is written as above. ECS uses an EC2 server to run a container, which can be a single or multiple EC2 servers for a project. A cluster is a set of EC2 for this project.
service
The Amazon ECS service allows you to run and maintain a specified number of instances of a task definition concurrently in an Amazon ECS cluster. If a task fails or stops for any reason, the Amazon ECS service scheduler launches another instance of the task definition to replace it and keep the service as many tasks as you need. In addition to maintaining the required number of tasks in the service, you can optionally run the service behind a load balancer. A load balancer distributes traffic among the tasks associated with a service.
In AWS Official, it is written as above. The concept of a service is very confusing, but the simple answer is to connect the container to the load balancer and manage which task definitions are adapted to the container. Regarding services, I think it will be easier to understand if you proceed with the settings while actually moving your hands.
task
A task is a collection of related containers. It's a little difficult to understand, so I'll explain it with a concrete example. There are often two container configurations for rails applications: web server + application server, but if either container is missing, the application will not start. Containers related to application startup in this way are called tasks on AWS. In AWS, it is necessary to set a task in order to start a container, which is called a task definition.
## 3. Actually deploy ECR / ECS on AWS * It is assumed that AWS VPC, ELB, RDS have been set and Dockerfile has been created in the development environment. Regarding AWS, [[Challenge AWS from 0] Deploy Rails apps to AWS using EC2 and VPC part1](https://qiita.com/shu1124/items/4a0337f3d9fa9227a705) Regarding Docker, [[Challenge Docker from 0] Build a development environment for nginx, puma, rails6.0, mysql using Docker](https://qiita.com/shu1124/items/ba370afcdfc99145e1dc) If you are unsure about AWS or Docker, please refer to it.
file organization
This time we will use an existing application. The file structure is as follows. Desktop/ ├ webapp/ ├ containers └ nginx └ Dockerfile └ nginx.conf ├ enviroment └ db.env ├ Dockerfile ├ docker-compose.yml ├ Gemfile ├ Gemfile.lock
Create a repository with ECR
If you search for ECR in the AWS menu, you will find ECR (* Actually, there is an ECR item in ECS). Then press Create Repository. Then, the following screen will be displayed, so enter any name you like for the repository name. Since the repository is required for Dockerfile, we will create two types, one for nginx and one for app. It is not necessary for the database because it uses RDS.
Next, the created repository will be displayed on the ECR menu, so select it and press the push command display in the upper right. Four commands will be displayed as shown below. Copy them and execute them in the terminal.
- The second command needs to specify the location of the Dockerfile, so this time it needs to be changed as follows.
Terminal
(for nginx)
docker build -t nginx -f ./container/nginx/Dockerfile .
(For app)
docker build -t app .
- If Amazon CLI is not installed, an error will occur. In that case, enter the following before executing.
Terminal
$ curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
$ unzip awscli-bundle.zip
$ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
This completes the ECR settings. The push command display command builds the image and pushes the image to the ECR. This means that your local Docker image is now stored on AWS.
Create a cluster with ECS
Cluster creation
Then select the ECS cluster from the menu on the left and click Create Cluster. Then the following screen will be displayed.
Select "EC2 Linux + Networking" here. Next comes the configuration details.
(* "AWS Fargate" will be described later)
I will explain only the part to be tampered with in the settings. --Set a descriptive cluster name. --Set the EC2 instance type to "t2.micro". Please note that if you select anything other than this, it will no longer be a free tier. --Set the number of instances to 1. ――Choose the key pair that you always use when you are logged in to EC2. If you keep the default, you will not be able to connect to EC2 created from ECS by SSH. --For networking, select the created VPC, for the subnet, select the public subnet in the VPC, and for the security group, select the security group used for the public subnet.
The other parts not mentioned are fine by default. This completes the cluster.
Task creation
Select a task definition from the menu on the left. The screen for selecting compatibility of the boot type will be displayed, so select EC2 (* Fargate will be described later). Then the following setting screen will appear.
This also explains only the setting tweaking.
--Enter a descriptive name for the task definition. --Select bridge as the network mode. By the way, this is the type of network used in the docker container --Click Add Volume at the bottom. The name should be easy to understand and the volume type should be Docker. The other two are OK with the default local and task. By the way, a volume is a place where Docker data is persisted. --Both task sizes are set to 512 this time. --Click Add Container in the middle. This time we need two containers, one for nginx and one for app. There are many places to enter here.
--Please enter a descriptive container name. --Enter the URI of ECR in the image. If it is nginx, enter the URI of the nginx image you pushed to ECR earlier. You can check the URI from the ECR menu. --Enter 0:80 for nginx and 0: 3000 for app for port mapping. * It is the same as the Docker port setting. --Only for app, but set environment variables. For example, if it is an RDS password, enter "RDS_PASSWORD" in Key and enter the password set in the value. Enter all the environment variables required for the app here. --Others are OK by default.
Now add two containers and click Create Task Definition. This completes the task.
Add service
Select the ECS cluster in the menu on the left and click the cluster you just created. The details screen will appear and the menu will be displayed at the bottom. Select a service from that menu and press the create button.
--Select EC2 as the startup type --For the task definition, select the task you created earlier. --Please give the service a descriptive name. --The number of tasks should be 2. This matches the number of containers. ――The rest is okay by default
After completing the above input, press the next step. Then the following screen will appear.
--Select "Application Load Balancer" for load balancing. Then, a place to select the load balancer name will appear, so let's select the created load balancer --Next, press Add "web: 0:80" (default) for the load balancing container to the load balancer. Then, the place to select the target group name will be displayed, so let's select the target group name created when creating the load balancer.
After completing the above input, press the next step. The settings will continue after that, but the defaults are OK. This completes the service.
Connect ECS with ssh ・ Access with browser
Now you're ready to go. Connect to ECS with ssh and check if there is a container. Select the cluster you created from the cluster in the ECS menu and click the ECS instance from the menu at the bottom. Since there is an item called EC2 instance, you can check EC2 created through ECS from there. Copy the EC2 private ip and enter the following in the terminal.
Terminal
$ ssh -i [Of the key pair] ec2-user@[Public IP]
(If successful, it will be displayed like this)
ast login: Sat Oct 24 10:28:34 2020 from 56.97.30.125.dy.iij4u.or.jp
__| __| __|
_| ( \__ \ Amazon Linux 2 (ECS Optimized)
____|\___|____/
For documentation, visit http://aws.amazon.com/documentation/ecs
After that, let's check the container and image with "docker ps" and [docker image]. You'll probably see what you've created so far. Finally, let's go inside the container, create a database, and migrate.
Terminal
$ docker exec -it [Container ID] bash
$ rails db:create
$ rails db:migrate
You should now be able to see your application in your browser. To check with a browser, select the load balancer from the EC2 menu and enter the DNS name of the load balancer specified when creating the service in the browser.
that's all.
- To put it simply, "AWS Fargate" is a mechanism in which AWS automatically handles the containers managed and operated by EC2. There will be a charge here, but the operating cost will be significantly reduced. Fargate is likely to become mainstream in the future. However, if it is an individual portfolio, I think that EC2 will make a big difference.
It's a review, but the general flow is as follows Register docker image in ECR → Create cluster (largest box) → Define task (define container relationship) → Create service (connect cluster and task, load balancer and container settings) → Complete
In the next article, we will connect ECS and CircleCI to actually build an automated deployment.
Summary / impression
I had a hard time because ECS was too unique, but when I actually handled it, I found it convenient. Since knowledge of AWS, Docker, and CircleCI is required, I would like to be able to set up automatic deployment while referring to the articles I have written so far.
reference
[Book] "Introduction to CircleCI Practice: Balancing Development Speed and Quality Brought by CI / CD Masato Urai (Author), Tomoya Otake (Author), Hirokuni Kim (Author)"
【qiita】 "I've just started CircleCI, so I've summarized it in an easy-to-understand manner" "[CircleCI] Introduced to Rails app (setting file)" "Even beginners can do it! Container deployment of Rails application with ECS x ECR x CircleCI "
[Other sites] "Automatically deploy to ECR / ECS using CircleCI Orbs"