① Push to GitHub ② Build runs on CircleCi ③ If it passes the test (Rspec), create a Docker image and push it to ECR ④ Using the latest Docker image, update the ECS task definition and complete the deployment
--People who have never touched ECS --People who want to move their hands and get a feel for the atmosphere --People who are creating a portfolio for job hunting
Steps to deploy a simple Rails app (just display "Hello World!") To AWS ECS. Please note that we do not take into consideration the subsequent operation, and it is only for reference.
There are probably various ways to do it depending on the person, so once you grasp the flow, please set it to your liking.
From the stage of preparing the application for deployment, everything is described in hands-on format (many screenshots), and basically it should work if you proceed as written.
There is a part to set arbitrary values (profile name, application name, etc.) at important points, so if you are uncertain, it may be better to unify with "sample-app" etc. like the author.
`* Although it is labeled for beginners, it does not explain each term because it is devoted to the purpose of" first actually moving your hands to get a feel for the atmosphere. " ``
* The concept is that even a super beginner on AWS can deploy like that if you follow the instructions for the time being. It is recommended that theoretical students learn the concept of ECR / ECS in other articles before entering. (Because I remember it with my body ...)
`* AWS resource names are basically arbitrary, so you can use them as you like (however, you need to match the description in the CircleCi config or when you type commands with ecs-cli). Also, for the parts that are not mentioned in particular, I think that the default state or blanks are okay for the time being. ``
* The revision number may vary due to the timing of taking screenshots, but please ignore it (sweat)
--Language: Ruby2.6 --Framework: Rails6 --Database: MySQL5.7 --Application server: Puma --Web server: Nginx
First, prepare a simple Rails app for deploying to ECS.
https://github.com/kazama1209/sample-app
$ git clone https://github.com/kazama1209/sample-app.git
$ cd sample-app
$ docker-compose build
$ docker-compose run web bundle exec rails webpacker:install
#If an error occurs with the ↑ command
$ docker-compose run web yarn install --check-files
$ docker-compose up -d
$ docker-compose run web bundle exec rails db:create
If you cloned the repository from ↑, config / master.key should not exist. If there is no master.key, an error will occur at the time of deployment, so generate it here according to the following procedure.
$ rm config/credentials.yml.enc
$ docker-compose run -e EDITOR=vim web rails credentials:edit
reference: I want to edit credentials with Rails on Docker ActiveSupport::MessageEncryptor::InvalidMessage
If you access http: // localhost
and the usual screen is displayed, it's OK.
Create a top page that returns "Hello World!", Which is the final goal of this time.
ruby:./app/controllers/home_controller.rb
class HomeController < ApplicationController
def index
end
end
erb:./app/views/home/index.html.erb
<h1>Hello World!</h1>
ruby:./config/routes.rb
Rails.application.routes.draw do
root 'home#index'
end
When "Hello World!" Is returned, the sample application is ready.
Now that the app is ready, let's deploy it to ECS.
This time, we will use the following two tools to deploy to ECS.
$ brew install awscli
$ brew install amazon-ecs-cli
Set up aws configure to use the above tools.
From the AWS console, select Services → IAM and click “Add User”.
Enter an arbitrary user name ("sample-app" this time), check "Access by program", and proceed to the next step.
From "Attach existing policy directly", attach the following two policies to the next step.
Ignore the tags this time and go to the next step.
Finally, the confirmation screen of the input information is displayed, so if there is no problem, click "Create user".
If you succeed in creating a user, you will be issued two "access key" and "secret access key", so take notes or download the csv file and keep it in a safe place.
$ aws configure --profile <The IAM user name you created earlier (this time "sample"-app」)>
AWS Access Key ID #The access key you created earlier
AWS Secret Access Key #The secret access key you created earlier
Default region name # ap-northeast-1
Default output format # json
Enter each as above.
When I created the IAM user earlier
I have attached two policies, but this alone will cause a permission error in the tool called "ecs-cli" that I will use later, so I have to add it separately here.
From the AWS console, select Services → IAM → Policies and click Create Policy.
Open the JSON tab and write the following.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"iam:AttachRolePolicy",
"iam:AddRoleToInstanceProfile",
"iam:CreateInstanceProfile",
"iam:CreateRole",
"iam:DeleteInstanceProfile",
"iam:DeleteRole",
"iam:DetachRolePolicy",
"iam:PassRole",
"iam:RemoveRoleFromInstanceProfile",
"ec2:DeleteInternetGateway",
"ec2:DeleteSecurityGroup",
"ec2:DeleteRouteTable"
],
"Resource": "*"
}
]
}
Enter a name and description for the policy and click "Create Policy".
From the AWS console, select Services → IAM → User and click “Add Access”.
Select the policy created earlier from "Attach existing policy" and add access authority.
Create a key pair as you will need it later when you enter EC2.
From the AWS console, select Services → EC2 → Key Pair and click “Create Key Pair”.
Enter the name and file format and click "Create Key Pair".
$ mv Downloads/sample-app.pem .ssh/
$ chmod 600 ~/.ssh/sample-app.pem
When completed, a ".pem" format file will be downloaded, so move it to the ".ssh" directory and change the permissions.
It is possible to create it manually from the console, but since it is necessary to create vpc and subnet together, this time I will create it collectively with ecs-cli.
Execute the following command.
$ ecs-cli configure profile --profile-name <Any profile name> --access-key <The access key you created earlier> --secret-key <The secret access key you created earlier>
$ ecs-cli configure --cluster <Any cluster name> --default-launch-type EC2 --config-name <Arbitrary setting name> --region ap-northeast-1
$ ecs-cli up --keypair <The key pair you created earlier> --capability-iam --size 2 --instance-type t2.samll --cluster-config <Arbitrary setting name> --ecs-profile <Any profile name>
Each value such as a key is different. In the case of the author, it looks like the following.
$ ecs-cli configure profile --profile-name sample-app --access-key ******************** --secret-key ****************************************
$ ecs-cli configure --cluster sample-app-cluster --default-launch-type EC2 --config-name sample-app-cluster --region ap-northeast-1
$ ecs-cli up --keypair sample-app --capability-iam --size 2 --instance-type t2.small --cluster-config sample-app-cluster --ecs-profile sample-app
INFO[0006] Using recommended Amazon Linux 2 AMI with ECS Agent 1.44.3 and Docker version 19.03.6-ce
INFO[0007] Created cluster cluster=sample-app-cluster region=ap-northeast-1
INFO[0009] Waiting for your cluster resources to be created...
INFO[0009] Cloudformation stack status stackStatus=CREATE_IN_PROGRESS
INFO[0070] Cloudformation stack status stackStatus=CREATE_IN_PROGRESS
INFO[0131] Cloudformation stack status stackStatus=CREATE_IN_PROGRESS
VPC created: vpc-*****************
Security Group created: sg-*****************
Subnet created: subnet-*****************
Subnet created: subnet-*****************
Cluster creation succeeded.
If all goes well, VPCs, security groups, subnets, etc. for the cluster will be created automatically as shown in ↑.
Select Service-> Elastic Container Service-> Cluster from the AWS console, and if it is created successfully, it is successful.
Create an RDS to use as a database.
Select Services → RDS from the AWS console and click “Create Database”.
--Creation method: Standard creation --Engine type: MySQL --DB instance size: Free tier
--DB instance identifier: sample-app-db --Master username: root --Password: password
`* This area is all optional. ``
--vpc: The vpc you just created --Subnet group: Create a new DB subnet group --Public access: Yes
`* Since this is just a practice, I have selected" Yes ", but in a production environment it may be better to set it to" No ". ``
First database name: sample_app_production
* When using the sample (sample-app) prepared at the beginning of the article, * Since the database name is defined in" config / database.yml "of sample-app, it is fixed to" sample_app_production "
`* Leave blank or default for parts not touched. ``
If there is no problem, click "Create Database".
↑ Success if it is created like this.
You also need to set up a security group, so click the link under "VPC Security Group".
Set as follows from "Edit inbound rule".
--Type: MYSQL / Aurora --Protocol: TCP --Port range --Source: 0.0.0.0/0
$ mysql -h <RDS endpoint> -u <RDS username> -p
Try hitting the ↑ command in the terminal, and if you can connect, you are successful.
From the AWS console, select Services → EC2 → Load Balancer and click “Create Load Balancer”.
There are three types, but select "Application Load Balancer".
--Name: sample-app-alb * Optional --Listener: OK as it is --VPC: Select the one that was automatically created earlier --subnet: Same as above
If you proceed, the security group setting screen will appear, so create an appropriate security group from "Create a new security group".
Target group settings.
--Target group: New target group --Name: sample-app-alb-tg * Optional
Register the EC2 that was automatically created when creating the cluster, and if there are no problems from the confirmation screen, click "Create" to complete.
From the AWS console, select Services → Amazon Elastic Container Registry and click “Create Repository”.
Enter an appropriate repository name for each and click "Create Repository".
Display the push command and execute the four commands in order from the top as written.
#Rails (use production Dockerfile)
$ docker build -f ./prod.Dockerfile . -t sample-app-rails
# Nginx
$ cd containers/nginx
$ docker build -f ./Dockerfile . -t sample-app-nginx
$ touch prod.Dockerfile
dockerfile:./prod.Dockerfile
FROM ruby:2.6.6
ENV LANG C.UTF-8
RUN apt-get update -qq && apt-get install -y build-essential libpq-dev nodejs
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash - && \
apt-get install nodejs
RUN apt-get update && apt-get install -y curl apt-transport-https wget && \
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list && \
apt-get update && apt-get install -y yarn
RUN mkdir /sample-app
WORKDIR /sample-app
ADD Gemfile /sample-app/Gemfile
ADD Gemfile.lock /sample-app/Gemfile.lock
RUN gem install bundler:2.1.4
RUN bundle install
ADD . /sample-app
#Preparing to communicate with Nginx
RUN mkdir -p tmp/sockets
VOLUME /sample-app/public
VOLUME /sample-app/tmp
RUN yarn install --check-files
RUN SECRET_KEY_BASE=placeholder bundle exec rails assets:precompile
It's basically the same as the one for development, but the last few lines are preparing to communicate with Nginx.
After typing everything, check the repository, and if the image is added, it is successful.
Create a task based on the image you pushed earlier.
$ mkdir ecs
$ touch ecs/docker-compose.yml
yml:./ecs/docker-compose.yml
version: 2
services:
app:
image: #ECR repository URI (Rails)
command: bash -c "bundle exec rails db:migrate && bundle exec rails assets:precompile && bundle exec puma -C config/puma.rb"
environment: #It's a practice, so it's a direct writing style, but I think it's actually better to use dotenv.
RAILS_ENV: production
RAILS_MASTER_KEY: # config/master.key value
DATABASE_NAME: sample_app_production
DATABASE_USERNAME: root
DATABASE_PASSWORD: password
DATABASE_HOST: #RDS endpoint
TZ: Japan
working_dir: /sample-app
logging:
driver: awslogs
options:
awslogs-region: ap-northeast-1
awslogs-group: sample-app-production/app
awslogs-stream-prefix: sample-app-production
nginx:
image: #ECR repository URI (Nginx)
ports:
- 80:80
links:
- app
volumes_from:
- app
working_dir: /sample-app
logging:
driver: awslogs
options:
awslogs-region: ap-northeast-1
awslogs-group: sample-app-production/nginx
awslogs-stream-prefix: sample-app-production
Execute the following command.
$ ecs-cli compose --project-name sample-app-task -f ./ecs/docker-compose.yml up --create-log-groups --cluster-config sample-app-cluster --ecs-profile sample-app
If successful, a "1" will be displayed for the running task.
Finally, paste the DNS name of the load balancer into the URL to access it.
Success if "Hello World!" Is displayed in the same way as the one created in the local environment.
The app works with just the cluster and the task, but if you create something called a "service" in the middle, it will restart when the container stops, or it will autoscale through the load balancer, so I created it because it is convenient. Keep it.
From the AWS console, click Services → Amazon Elastic Container Service → Cluster Name to open the Services tab and go to the Create page.
--Boot type: EC2 --Task definition: sample-app-task * The one created earlier --Cluster: sample-app-cluster * Same as above --Service name: sample-app-service * Optional --Other: As shown in the image
--Load balancer type: Application Load Balancer --Load balancer name: What you created earlier --Other: As shown in the image
--Target group name: What you created earlier --Other: As shown in the image
Finally, a confirmation screen will be displayed, so if there are no problems, click Create.
If it is created successfully, it is completed.
At this rate, two tasks (one started from the terminal with ecs-cli and the other started by creating a service) are running, so the former is stopped and OK. ..
If this is left as it is, every time there is a change, it will be necessary to manually perform troublesome work such as "build → push → task redefinition", so a common mechanism such as "push to CircleCi → build & test → automatic deployment to ECR / ECS" Will continue to build.
First, install Rspec for testing before deployment.
./Gemfile
group :development, :test do
gem 'rspec-rails'
end
#I updated the Gemfile, so build again
$ docker-compose build
$ docker-compose run web bundle exec rails generate rspec:install
create .rspec
create spec
create spec/spec_helper.rb
create spec/rails_helper.rb
ruby:./.rspec
--format documentation
If you add one line of ↑, it will be easier to see the output display when Rspec is executed.
ruby:./spec/rails_helper.rb
Dir[Rails.root.join('spec', 'support', '**', '*.rb')].sort.each { |f| require f }
Although it is not essential, if you decide to create a helper method for testing later, "spec / support" will be used as the file storage, so set it for the time being.
By default, it is commented out, so you can remove it.
ruby:./config/application.rb
config.generators do |g|
g.test_framework :rspec,
view_specs: false,
helper_specs: false,
controller_specs: false,
routing_specs: false
end
If this is left as it is, various test files will be created automatically when you type the rails g
command, so if you do not want to create extra ones, set it in" config / application.rb ".
$ docker-compose run web bundle exec rspec
No examples found.
Finished in 0.00276 seconds (files took 0.12693 seconds to load)
0 examples, 0 failures
I haven't written any tests yet, so of course this is the case.
As a starting point, let's write a Request Spec to check if a normal response is returned to the request.
$ docker-compose run web bundle exec rails g rspec:request home
create spec/requests/homes_spec.rb
ruby:./spec/requests/home_spec.rb
require 'rails_helper'
RSpec.describe "Home", type: :request do
describe "GET /" do
it "works successfully" do
get root_path
expect(response).to have_http_status(200)
end
end
end
A test to see if the status code 200 is returned when accessing "/".
$ docker-compose run web bundle exec rspec
Home
GET /
works successfully
Finished in 0.53664 seconds (files took 8.4 seconds to load)
1 example, 0 failures
Execute rspec again, and if it passes without any problem, it succeeds.
Next, make the settings to actually link with CircleCi.
./Gemfile
group :development, :test do
gem 'database_cleaner'
gem 'rspec_junit_formatter'
gem 'webdrivers', '~> 3.0'
end
#I updated the Gemfile, so build again
$ docker-compose build
$ mkdir .circleci
$ touch .circleci/config.yml
$ touch config/database.yml.ci
$ docker-compose run web bundle exec rails db:schema:dump
yml:./.circleci/config.yml
version: 2
jobs:
build:
docker:
- image: circleci/ruby:2.6.6-node-browsers
environment:
- BUNDLER_VERSION: 2.1.4
- RAILS_ENV: 'test'
- image: circleci/mysql:5.7
environment:
- MYSQL_ALLOW_EMPTY_PASSWORD: 'true'
- MYSQL_ROOT_HOST: '127.0.0.1'
working_directory: ~/sample_app
steps:
- checkout
- restore_cache:
keys:
- v1-dependencies-{{ checksum "Gemfile.lock" }}
- v1-dependencies-
- run:
name: install dependencies
command: |
gem install bundler -v 2.1.4
bundle install --jobs=4 --retry=3 --path vendor/bundle
- save_cache:
paths:
- ./vendor/bundle
key: v1-dependencies-{{ checksum "Gemfile.lock" }}
# database setup
- run: mv ./config/database.yml.ci ./config/database.yml
# database setup
- run:
name: setup database
command: |
bundle exec rake db:create
bundle exec rake db:schema:load
# install yarn
- run:
name: install yarn
command: yarn install
# install webpack
- run:
name: install webpack
command: bundle exec bin/webpack
# run tests
- run:
name: run rspec
command: |
mkdir /tmp/test-results
TEST_FILES="$(circleci tests glob "spec/**/*_spec.rb" | \
circleci tests split --split-by=timings)"
bundle exec rspec \
--format progress \
--format RspecJunitFormatter \
--out /tmp/test-results/rspec.xml \
--format progress \
$TEST_FILES
# collect reports
- store_test_results:
path: /tmp/test-results
- store_artifacts:
path: /tmp/test-results
destination: test-results
yml:./config/database.yml.ci
test:
adapter: mysql2
encoding: utf8
pool: 5
username: 'root'
port: 3306
host: '127.0.0.1'
database: sample_app_test
ruby:./spec/rails_helper.rb
RSpec.configure do |config|
# config DataBaseCleaner
config.before(:suite) do
DatabaseCleaner.strategy = :transaction
DatabaseCleaner.clean_with(:truncation)
Rails.application.load_seed
end
config.before(:each) do
DatabaseCleaner.start
end
config.after(:each) do
DatabaseCleaner.clean
end
end
ruby:./db/schema.rb
# This file is auto-generated from the current state of the database. Instead
# of editing this file, please use the migrations feature of Active Record to
# incrementally modify your database, and then regenerate this schema definition.
#
# This file is the source Rails uses to define your schema when running `rails
# db:schema:load`. When creating a new database, `rails db:schema:load` tends to
# be faster and is potentially less error prone than running all of your
# migrations from scratch. Old migrations may fail to apply correctly if those
# migrations use external dependencies or application code.
#
# It's strongly recommended that you check this file into your version control system.
ActiveRecord::Schema.define(version: 0) do
end
https://app.circleci.com/projects/project-dashboard/github/GitHubのアカウント名/
↑ Find the repository you want to link from the CircleCi dashboard and click "Set Up Project". Set according to the instructions displayed on the screen.
Now, when you make a new push to GitHub in the future, build & test will run automatically based on the contents written in ".circleci / config.yml".
If there is no problem, "SUCCESS" should be displayed. This completes the initial settings.
Using Orb added from version 2.1 of CircleCi, when changes are made to the master branch, build & test with CircleCi, automatically create an image and push it to ECR, update the ECS service and update the task Try to redefine.
Register the environment variables required for deployment from the CircleCi setting screen in advance.
yml:./.circleci/config.yml
version: 2.1
orbs:
aws-ecr: circleci/[email protected]
aws-ecs: circleci/[email protected]
jobs:
test:
docker:
- image: circleci/ruby:2.6.6-node-browsers
environment:
- BUNDLER_VERSION: 2.1.4
- RAILS_ENV: 'test'
- image: circleci/mysql:5.7
environment:
- MYSQL_ALLOW_EMPTY_PASSWORD: 'true'
- MYSQL_ROOT_HOST: '127.0.0.1'
working_directory: ~/project
steps:
- checkout
- restore_cache:
keys:
- v1-dependencies-{{ checksum "Gemfile.lock" }}
- v1-dependencies-
- run:
name: install dependencies
command: |
gem install bundler -v 2.1.4
bundle install --jobs=4 --retry=3 --path vendor/bundle
- save_cache:
paths:
- ./vendor/bundle
key: v1-dependencies-{{ checksum "Gemfile.lock" }}
- run: mv ./config/database.yml.ci ./config/database.yml
- run:
name: setup database
command: |
bundle exec rake db:create
bundle exec rake db:schema:load
- run:
name: install yarn
command: yarn install
- run:
name: install webpack
command: bundle exec bin/webpack
- run:
name: run rspec
command: |
mkdir /tmp/test-results
TEST_FILES="$(circleci tests glob "spec/**/*_spec.rb" | \
circleci tests split --split-by=timings)"
bundle exec rspec \
--format progress \
--format RspecJunitFormatter \
--out /tmp/test-results/rspec.xml \
--format progress \
$TEST_FILES
- store_test_results:
path: /tmp/test-results
- store_artifacts:
path: /tmp/test-results
destination: test-results
workflows:
version: 2
test_and_deploy:
jobs:
- test
#Push the image to ECR after building
- aws-ecr/build-and-push-image:
requires:
- test
account-url: AWS_ECR_ACCOUNT_URL
region: AWS_REGION
aws-access-key-id: AWS_ACCESS_KEY_ID
aws-secret-access-key: AWS_SECRET_ACCESS_KEY
create-repo: true
dockerfile: ./prod.Dockerfile
repo: "${MY_APP_PREFIX}-rails"
tag: "${CIRCLE_SHA1}"
filters:
branches:
only:
- master
#Update ECS services to redefine tasks
- aws-ecs/deploy-service-update:
requires:
- aws-ecr/build-and-push-image
family: "${MY_APP_PREFIX}-task"
cluster-name: "${MY_APP_PREFIX}-cluster"
service-name: "${MY_APP_PREFIX}-service"
container-image-name-updates: "container=app,tag=${CIRCLE_SHA1}"
Make sure that each environment variable is correct.
erb:./app/views/home/index.html.erb
<h1>Hello World!</h1>
<p>Completed auto deploy with CircleCi</p>
Let's change the top page a little so that it is easy to see if the automatic deployment worked.
When I actually push it to the master branch and make changes, the jobs with deployment start running on CircleCi.
`* Please note that it takes about 10 to 15 minutes to complete all the flows. ``
Access the DNS name of the load balancer again, and if the previous change is updated properly, it is successful.
`* It will take some time before it is reflected, so wait patiently. ``
Two tasks, an old task and a new task, are being executed, but the old one will be erased by itself over time (thanks to the service?), So you can leave it as it is.
Thank you for your hard work.
I'm still studying, so I'll update it whenever I have something.
Currently, the work done while moving hands on the console and the work done by hitting commands on the terminal are mixed up, so I would like to unify all of them to the latter if possible.
I want to be able to do it in one shot using Terraform.
I would be grateful if you could comment if there is something stuck or something like this is better.
--Recommended "t2.small" or above. I think that "t2.micro" is the standard when using it in the free frame, but in the case of the author, there was a problem that the memory became insufficient with t2.micro.
service sample-app-service was unable to place a task because no container instance met all of its requirements. The closest matching container-instance c4b1a3e7-3209-408c-9501-7b3ea30f97f7 has insufficient memory available. For more information, see the Troubleshooting section.
Reference: https://aws.amazon.com/jp/premiumsupport/knowledge-center/ecs-container-instance-cpu-error/
In this article, the command when creating a cluster
$ ecs-cli up --keypair sample-app --capability-iam --size 2 --instance-type t2.small --cluster-config sample-app-cluster --ecs-profile sample-app
↑ The instance type is specified in this part.
--There are some parts of the nginx.conf file that do not work properly unless you change them, so you need to modify them as appropriate while googled.
If the connection between the app container and the nginx container doesn't work
2020/09/13 20:02:57 [crit] 7#7: *456 connect() to unix:///sample-app/tmp/sockets/puma.sock failed (2: No such file or directory) while connecting to upstream, client: *********, server: localhost, request: "GET / HTTP/1.1", upstream: "http://unix:///sample-app/tmp/sockets/puma.sock:/500.html", host: "***********"
↑ I am annoyed endlessly with an error like this.
--It is better to check many times whether the values of the environment variables registered in CircleCi are correct. It takes a lot of time to build, so a single failure is a huge waste of time.
Recommended Posts