[JAVA] About testing web applications using DynamoDB and automating deployment to Fargate

Introduction

• The screen is Java (Spring Boot) + DynamoDB, Web application, client tool is Go, API part is API Gateway + Lambda (Python), and it is built with a fairly unconventional configuration.
• This time, I would like to introduce a part of how to automate the testing and deployment of Web applications using SpringBoot + DynamoDB in that architecture.
• I wonder if there is a reference part even if you are not a Java engineer. If you like, please read just the summary of the article.

Article summary

What are you automating

What kind of technology are you using

• ** WEB application **
  • SpringBoot 1.5.6 + Thymeleaf
• DB uses only DynamoDB. I implemented the DynamoDB access framework myself.
• DynamoDBLocal
• You can start DynamoDB locally.
• Since it can be started in-memory, it is good that you do not have to clean up the data by starting it with CI and dropping it at the end.
• Selenide
• Selenium wrapper library. I like this place and use it.
• Can be written with CSS Selector (the amount of description is reduced anyway)
• Test code and CSS Selector description can be separated quickly
• You can easily switch drivers etc. with environment variables. (Some are implemented by myself)
• No need for detailed wait when loading the screen, screen capture when the test fails, or the part that was expanded by yourself
• ** Browser for E2E testing **
• When I saw the news that the main committer of PhantomJS was missing, I took the plunge and started using ** Chrome Headless **.
• Reference: Phantom.js maintainer, questioning the future of the project and stepping down
• The chrome driver uses 2.33, but it can be used without any problems.
• I'll come back to it later, but ** good, everyone is too lazy to run chrome headless on Amazon Linux, so let's stop. ** **
• ** OS image used in the container **
• I thought I couldn't use CentOS because of the long build time, so I switched to ** Alpine Linux **.
• The development environment is based on openjdk alpine linux and the locale is set.
• I also included the log driver settings (fluentd), but since it is a development environment, I am using CloudWatch Logs, which Fargate sets automatically.
• ** Docker repository **
• ** Use ECR **
• Docker Hub can only have one private repository. It doesn't do automation build.
• I also set up artifactory on EC2, but when I thought that I had to operate it, I quit because it was a bit silly. I did.
• The deciding factor is that if you use these external containers, you have to manage ID / PW, but AWS has a wonderful mechanism called STS, so I decided that it is better to get on.
• ** Container orchestration software **
• ** I started using AWS Fargate **.
• I used to use ECS (EC2) on Spot Fleet until last month, but I lost my curiosity to use Fargate. (Of course, the price is higher for Fargate)
• ECS (Fargate) runs on us-east-1.
• I'm trying to use ap-northeast-1 in production, but DynamoDB doesn't have a concept like schema, so I can't put the same table in the same region. But I think it's okay to touch Fargate because of this.
• ** Deployment script to ECS (Fargate) **
• At the time of ECS (EC2), I used to deploy using silinternational / ecs-deploy, but I forked because it did not support Fargate. I am using it after fixing it.
• ecs-deploy is just a shell, so it's convenient to carry. Click here for Fargate compatible version. * AWS Fargate BlueGreenDeployment * https://github.com/uzresk/ecs-deploy.git

How do you do it

• When I run these tasks from Jenkins, I wrote it in the shell at first, but it's more mendokusai than I expected.
• I've done a lot of trial and error, but ** Except for deploying to Fargate, you can just do mvn install. ** **

Practical edition

Use DynamoDB locally

• To use DynamoDB locally, use the official module DynamoDB Local.
• I use it floating on Docker. If you google this page, various articles will appear, and there are no particular points to be addicted to, so I will omit it.
• By the way, GUI is also included, but I don't use it because I have to write it in node.

Automate tests to access DynamoDB

• Looking at the AWS manual of DynamoDBLocal, it seems that it can be operated in-memory like H2. .. ** **
• If you can move it in-memory, you will not forget to erase the data, so there is no way not to use this.
• I wondered if I could move it with this procedure.
• 1. Start DynamoDB in-memory
• 2. Create a table
• 3. Run the test
• 4. Stop DynamoDB (data disappears)
• Since the environment to be used is a mixture of Windows and Linux, I thought that I would like to avoid forcibly doing something with shell or bat to start or create a table definition.
• I started making maven plugins quietly, but it's too dull. So that's it. ..

jcabi-dynamodb-maven-plugin

• You can do this.
• DynamoDB Local can be started by pre-integration task
• You can execute the specified table definition script after starting DynamoDB Local.
• You can stop DynamoDB Local with the post-integration task
• Various parameters can be set when DynamoDB is started (starting location, inMemory mode, port number, etc.)
• It seems that you can use it for personal or commercial use if you display the license and copyright

Flow of downloading, starting, inputting DDL, testing, and stopping DynamoDB Local

• If you explain in a little more detail, the flow will be like this.
• 1. Download DynamoDB Local with maven-dependency-plugin and place it in dynamod b-dist
• 2. Start DynamoDB from dynamodb-dist on the port specified by $ {dynamodb.port} using jcabi-dynamodb-maven-plugin
• 3. After startup, the table creation script written in <tables> will run.
• If you want json at the table quickly, please use this
• Reference: One liner to convert existing DynamoDB table to json format

pom.xml

<plugins>
    <plugin>
        <artifactId>maven-dependency-plugin</artifactId>
        <version>2.8</version>
        <executions>
            <execution>
                <id>unpack-dynamodb-local</id>
                <goals>
                    <goal>unpack</goal>
                </goals>
                <configuration>
                    <artifactItems>
                        <artifactItem>
                            <groupId>com.jcabi</groupId>
                            <artifactId>DynamoDBLocal</artifactId>
                            <version>2015-07-16</version>
                            <type>zip</type>
                            <outputDirectory>${project.build.directory}/dynamodb-dist</outputDirectory>
                            <overWrite>false</overWrite>
                        </artifactItem>
                    </artifactItems>
                </configuration>
            </execution>
        </executions>
    </plugin>
    <plugin>
        <groupId>com.jcabi</groupId>
        <artifactId>jcabi-dynamodb-maven-plugin</artifactId>
        <version>0.9.1</version>
        <executions>
            <execution>
                <goals>
                    <goal>start</goal>
                    <goal>create-tables</goal>
                    <goal>stop</goal>
                </goals>
                <configuration>
                    <port>${dynamodb.port}</port>
                    <dist>${project.build.directory}/dynamodb-dist</dist>
                    <arguments>
                        <argument>-inMemory</argument>
                        <argument>-sharedDb</argument>
                    </arguments>
                    <tables>
                        <table>${basedir}/src/main/resources/ddl/Asset.json</table>
                        <table>${basedir}/src/main/resources/ddl/AssetGroup.json</table>
                        <table>${basedir}/src/main/resources/ddl/AssetRole.json</table>
                        <table>${basedir}/src/main/resources/ddl/Contract.json</table>
                        <table>${basedir}/src/main/resources/ddl/Invitation.json</table>
                        <table>${basedir}/src/main/resources/ddl/Member.json</table>
                        <table>${basedir}/src/main/resources/ddl/Organization.json</table>
                        <table>${basedir}/src/main/resources/ddl/OrgKey.json</table>
                        <table>${basedir}/src/main/resources/ddl/OrgMember.json</table>
                        <table>${basedir}/src/main/resources/ddl/Role.json</table>
                        <table>${basedir}/src/main/resources/ddl/Service.json</table>
                        <table>${basedir}/src/main/resources/ddl/Tag.json</table>
                    </tables>
                </configuration>
            </execution>
        </executions>
    </plugin>
<plugins>

• Since it works with the integration task, you can start-test-stop DynamoDB by executing the following command.

mvn verify

• integration test only works with classes named IT by default. I didn't like to name the test class IT, so I use Category to specify it.
• Reference: junit-team / junit4 / Categories

E2E test automation

• I want to do E2E test with the flow of integration test.
• I use Selenide. I like this place.
• Settings can be set in system properties (Configuration)
• Headless switching was not possible in the system property settings, but it has been slightly improved so that it can be done.
• If you import statically, you can implement it with CSS Selector like $ ("# id "), so the amount of description is drastically reduced compared to Selenium.
• I like the Page pattern
• You can expel the CSSSelector description from the JUnit test class, so it's easy to understand what you're testing.

Launch the Spring Boot application

• If you use spring-boot-maven-plugin, it will start with pre-integration and stop with post-integration. You can easily do that.

pom.xml

<plugins>
    <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <configuration>
            <mainClass>jp.gr.java_conf.uzresk.puzzle.web.WebApplication</mainClass>
            <profiles>
                <!-- application-integration.Launch the application using yml-->
                <profile>integration</profile>
            </profiles>
        </configuration>
        <executions>
            <execution>
                <id>pre-integration-test</id>
                <goals>
                    <goal>start</goal>
                </goals>
            </execution>
            <execution>
                <id>post-integration-test</id>
                <goals>
                    <goal>stop</goal>
                </goals>
            </execution>
        </executions>
    </plugin>

• I also prepared a profile for integration, and in the case of integration, I set it to start on port 18080.

application-integration.yml

server:
  port: 18080
settings:
  dynamodb:
    host: localhost
    port: 10000
    region: ap-northeast-1

Absorb the environmental difference between the local environment and CI

• If you write and execute Selenide code in JUnit, it will work, but there are differences in the environment.

• In @BeforeClass before moving Selenide, various settings can be made with -D (system property).

    @BeforeClass
    public static void beforeClass() {

        //Change the base URL(Default:http://localhost:8080)
        String webApplicationPort = System.getProperty("webapp.port", "8080");
        Configuration.baseUrl = String.format("http://localhost:%s/app/",webApplicationPort);

        //Make the default driver Chrome Driver
        Configuration.browser = System.getProperty("selenide.browser", WebDriverRunner.CHROME);
        System.setProperty("webdriver.chromedriver", System.getProperty("webdriver.chromedriver", "./drivers/chromedriver.exe"));

        // headless
        Configuration.headless = Boolean.parseBoolean(System.getProperty("selenide.headless", "false"));
    }

• The integration test is set by maven-failsafe-plugin, but here the port, ChromeDriver location, and Headless or not are switched.

pom.xml

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>2.11</version>
        <configuration>
            <includes>
                <include>**/*.class</include>
            </includes>
            <groups>
                jp.gr.java_conf.uzresk.puzzle.web.test.IntegrationTest
            </groups>
            <summaryFile>${report.dir}/failsafe-summary.xml</summaryFile>
            <reportsDirectory>${report.dir}</reportsDirectory>
            <!--Settings for CI-->
            <systemPropertyVariables>
                <webapp.port>18080</webapp.port>
                <selenide.headless>true</selenide.headless>
                <webdriver.chromedriver>./drivers/chromedriver</webdriver.chromedriver>
            </systemPropertyVariables>
        </configuration>
        <executions>
            <execution>
                <goals>
                    <goal>integration-test</goal>
                    <goal>verify</goal>
                </goals>
            </execution>
        </executions>
    </plugin>
<plugins>

Install Chrome on Amazon Linux

• Actually, this took the longest time.
• Since Amazon Linux is an OS that requires CUI, it does not contain any GUI modules, and it cannot be obtained from Amazon's yum repository, so it can be pulled from another location to resolve dependencies one by one. I have to do it. This cannot be resolved by resolving the dependency of that module. It took a lot of time to repeat.
• ** The conclusion is ... **
• ** I'm addicted to trying to run chrome on Amazon Linux. ** **
• ** Use Debian or CentOS7 / Redhat7. ** **
• ** Even if you are using Amazon Linux, if you are using the cloud, set Jenkins Slave to CentOS 7 and run JOB. ** **
• If you still want to try it, please refer to this page.
  • INSTALLING GOOGLE CHROME ON CENTOS, AMAZON LINUX, OR RHEL
• The packaged Chrome featured here is old, so you have to go the Hard Way after all.
  • Having trouble installing Google Chrome -- where can I find dependencies?
• You need an AWS account.

By the way, I started DynamoDB locally and tested it, and I was able to test around the screen. Let's deploy it.

Deploy

Prepare a Dockerfile

• Prepare a Dockerfile. I think it's good to refer to the official blog of Spring Boot, but I think that there is no problem if you use at least this much.
  • Spring Boot with docker

FROM openjdk:8u131-jdk-alpine

LABEL maintainer "uzresk"

RUN apk --update add tzdata && \
    cp /usr/share/zoneinfo/Asia/Tokyo /etc/localtime && \
    apk del tzdata && \
    rm -rf /var/cache/apk/*

VOLUME /tmp

ARG JAR_FILE
ADD ${JAR_FILE} xxx-web.jar
ENTRYPOINT ["java","-jar","/xxx-web.jar"]

push to docker build and ECR

• Use spotify / dockerfile-maven to make a container using Dockerfile.
• Create a container with the jar created by build as an argument of Dockerfile, and push it to the location specified in <repository>.
• If you need to authenticate, you need to set it in settings.xml, but my environment has a role to push to ECR to EC2 instance running Jenkins, and docker login You don't have to set it because it is executed by Jenkins in advance.

            <!--Create a container from Dockerfile and deploy it to ECR--> 
            <plugin>
                <groupId>com.spotify</groupId>
                <artifactId>dockerfile-maven-plugin</artifactId>
                <version>1.3.6</version>
                <configuration>
                    <repository>xxxx.dkr.ecr.ap-northeast-1.amazonaws.com/xxx-web</repository>
                    <tag>${project.version}</tag>
                    <buildArgs>
                        <JAR_FILE>target/${project.build.finalName}.jar</JAR_FILE>
                    </buildArgs>
                </configuration>
                <executions>
                    <execution>
                        <id>default</id>
                        <phase>install</phase>
                        <goals>
                            <goal>build</goal>
                            <goal>push</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

Where it gets stuck

• Pushing to private registry fails with 'No basic auth' for older style config.json
• I sed after docker login with Jenkins and added https: // to .docker / config.json to avoid it.

Deploy to Fargate

• At the time of ECS (LaunchType EC2), I used a shell called ecs-deploy, but it does not support Fargate.
• Fork is made to support deployment to Fargate, so please use it if you like. Usage is exactly the same as ecs-deploy.
  • https://qiita.com/uzresk/items/02f378f31e6f496b41a0

Jenkins settings

• Let's do docker login as a preliminary preparation.
• By hitting aws ecs get-login ..., the docker login command will be returned, so it is executed as it is with eval.
• Don't forget that the --no-include-email option is required for docker 17.06 and later. * http://docs.aws.amazon.com/ja_jp/AmazonECR/latest/userguide/ECR_AWSCLI.html

docker logout xxxx.dkr.ecr.ap-northeast-1.amazonaws.com
eval $(aws ecr get-login --no-include-email --region ap-northeast-1)

# dockerfile-maven-config with plugin.https to json://Fail without
sed -i 's/xxxx/https:\/\/xxxx/' ~/.docker/config.json

• Run maven.

mvn install

• Run ecs-deploy

chmod +x $WORKSPACE/web/bin/ecs-deploy
$WORKSPACE/web/bin/ecs-deploy -c web-cluster -n web-service -i xxxx.dkr.ecr.ap-northeast-1.amazonaws.com/xxxxxx-web:0.0.1-SNAPSHOT -t 600 -r us-east-1

• Since it is newly tagged, delete the docker image that has been untagged.

docker images | grep "[image name]" | grep "<none>"  | awk '{print $3}' | xargs docker rmi

at the end

• I am running this cycle at the timing of pushing, but it takes about 3 minutes and 30 seconds to complete the test, so I can operate it without problems so far.
• I'm wondering if the E2E test part, which will take time if the app grows a little more, can be run with Jenkins Slave. It would be interesting if we could move this in parallel on Fargate.