[JAVA] I checked the automatic unit test creation tool (end of 2019 version)

I checked the automatic unit test creation tool (end of 2019 version)

Hello. 2019 is almost over. This article is the 25th day article of Software Testing Advent Calendar 2019.

The previous day's article was written by my colleague @ozhiro san, "What is the purpose of your automated testing? ](Https://qiita.com/ozhiro/items/5aa95c6360a8930df325). __be moved! __ Well, today I examined the current status of automatic unit test creation tools.

Why did you decide to look it up?

Unit testing is important, isn't it? As software is developed and modified on a daily basis, unit testing is a test tool that ensures the quality of the developed software. In addition, test execution can be automated, reducing test man-hours.

However, test code development is required to execute unit tests, and development man-hours are required accordingly. For more comprehensiveness, it can take as long as developing the software under test.

Against this background, there are not enough development man-hours and time periods, so it is quite common to do manual tests without creating unit tests. (It's different now, but the project of the previous job and the previous job was like that.) Also, I already have a huge amount of source code and I want to create a unit test from now on, but where should I start? I think there are many situations where you don't know.

I thought there. In such a world where AI and RPA (Robotic Process Automation) are booming now, I think that tools that analyze source code and automatically create unit tests already exist in this world. Isn't it like this in 2020? When.

We in 2020 (image): 2020年イメージ図

Find out what tools are available

I will immediately search the Internet for what kind of tools are available. The search condition is "generate unit test automatically", and since I am an engineer who mainly uses Java, I also add "Java" to the condition.

Surprisingly, there are few search results and old articles can be seen here and there, so I felt a little uneasy ... (´ ・ ω ・ `), but I picked up about four good ones.

I couldn't understand the usability just by looking at the outline, so I created a test code from the actual source code and evaluated it. We have prepared the following two types of source code.

  1. [Class whose processing is closed only in the class and has only stateless methods](https://github.com/policeman-kh/generate_test/blob/master/src/main/java/test/NumberUtils. java)
  2. Spring MVC service class

I wanted to add some branching to method 1 and add the test coverage rate to the evaluation. 2 is a class with a DI (Dependency Injection) instance. I want you to replace the DI instance with a mock of Mockito etc. in the test code. I have a feeling.

The concrete source code was prepared on GitHub as one project together with the created test code. There are links to GitHub in some places, so please see that for details.

Tool introduction and evaluation

Now, I would like to introduce each tool and evaluate the generated test code.


The generated test code (partial) is below. The DI instance is mocked using Mockito. __Sounds good! __ Since the test code is only generated according to the template, there is almost no coverage rate, and it is necessary to develop it separately based on the generated test code. However, the speed of automatic generation is a few seconds, and it was not bad compared to creating test code from scratch.

Rating: Speed: ★★★★★ (several seconds) coverage: Customization: ★★ (Test framework can be selected)

Click here for test code and how to generate

class ZipCodeServiceTest {
    ZipCodeApi zipCodeApi;
    ZipCodeService zipCodeService;

    void setUp() {

    void testSearch() throws IOException {

        List<ZipCodeData> result = zipCodeService.search("zipCode");
                        new ZipCodeData("zipcode", "prefcode", "address1", "address2", "address3", "kana1",
                                        "kana2", "kana3")), result);


Is it almost the same as TestMe? It's an impression, but you can edit the template on the setting screen. I want to use another library such as AssertJ or Hamcrest for assertions, so this area is easy to use. If it is not a paid license __, I would like to use it.

Rating: Speed: ★★★★★ (several seconds) coverage: Customization: ★★★★ (Select test framework, edit template)

Click here for test code and how to generate

class ZipCodeServiceTest {

    private ZipCodeApi mockZipCodeApi;

    private ZipCodeService zipCodeServiceUnderTest;

    void setUp() {
        zipCodeServiceUnderTest = new ZipCodeService(mockZipCodeApi);

    void testSearch() throws Exception {
        // Setup
        final List<ZipCodeData> expectedResult = Arrays.asList(
                new ZipCodeData("zipcode", "prefcode", "address1", "address2", "address3", "kana1", "kana2",

        // Run the test
        final List<ZipCodeData> result = zipCodeServiceUnderTest.search("zipCode");

        // Verify the results
        assertEquals(expectedResult, result);


Since the test code created from 2 was not good enough, the test code (part) of 1 is described below.

It takes 3 minutes to generate the test code. However, this area can be changed with the argument at the time of generation. Looking at the coverage rate of the test code ... __100% __ Amazing! EvoSuiteカバレッジ率

However, the generated test code is not readable and I don't feel like maintaining it manually later. Many annotations that I do not understand are given, and the point that it inherits the base class of EvoSuite is also a little unsatisfactory. For mock, take a look at 1 test code However, it was not set properly. This Issue says that it can be adjusted with arguments when generating test code, but an error occurs at runtime and it ends up being a mock. It was not possible to generate some test code. (Abandoned on the way)

However, the coverage rate is very good, so when creating a stateless and public API, it would be better to use it for finding bugs. I feel like.

Rating: Speed: ★★ (3 minutes) Coverage: ★★★★★ (100%!) Customization: ★★ (Some can be specified by arguments when generating test code)

Click here for test code and how to generate

@RunWith(EvoRunner.class) @EvoRunnerParameters(mockJVMNonDeterminism = true, useVFS = true, useVNET = true, resetStaticState = true, separateClassLoader = true, useJEE = true) 
public class NumberUtils_ESTest extends NumberUtils_ESTest_scaffolding {

  @Test(timeout = 4000)
  public void test00()  throws Throwable  {
      boolean boolean0 = NumberUtils.isOddNumberBetween1And50(50);
// omit other testcases.


What you can do is almost the same as Evosuite. Coverage was 100% as well. __ Good. __ It's not easy to use because there is only command line execution to generate test code, but I think it's good that runtime Jar is not required to execute the generated test code.

Rating: Speed: ★★ (a few minutes) Coverage: ★★★★★ (100%!) Customization: ★★ (Some can be specified by arguments when generating test code)

Click here for test code and how to generate

public class NumberUtilsTest {

    public static boolean debug = false;

    public void test01() throws Throwable {
        if (debug)
            System.out.format("%n%s%n", "RegressionTest0.test01");
        // The following exception was thrown during execution in test generation
        try {
            boolean boolean1 = test.NumberUtils.isEvenNumberBetween1And10WithBadCode((-1));
            org.junit.Assert.fail("Expected exception of type java.lang.IllegalArgumentException; message: input should be between 1 and 10.");
        } catch (java.lang.IllegalArgumentException e) {
        // Expected exception.


So far, we have investigated and evaluated the four unit test automatic creation tools. To be honest, I feel that it is far from the image I expected, but I think it was good to know the current situation.

The source code, test code, library set, and HowTo ReadMe used in this study are stored on the following GitHub. If you would like to create test code using these tools, we would appreciate it if you could refer to it.

Investigation of auto generation unit tests with Java

Well, the long Advent calendar is over today. Thank you to everyone, both readers and writers! __ Have a nice year!

Recommended Posts

I checked the automatic unit test creation tool (end of 2019 version)
Automatic creation of Java unit test result report
[CircleCI] I was addicted to the automatic test of CircleCI (rails + mysql) [Memo]
I checked the place of concern of java.net.URL # getPath
I checked the number of taxis with Ruby
Try the free version of Progate [Java I]
I made a Japanese version of Rails / devise automatic email
Implementation of unit test code
Switch the version of bundler
I made a tool to output the difference of CSV file
I was addicted to the API version min23 setting of registerTorchCallback
[Java version] The story of serialization
Check the version of Cent OS
I read the source of ArrayList I read
I read the source of Integer
I read the source of Long
Get the ID of automatic numbering
I read the source of Short
Significance of existence of unit test (self-discussion)
I read the source of Byte
I read the source of String
I checked the specification of cols attribute of AsciiDoc table & internal implementation of Asciidoctor
I get got errors: User must exist in the unit model test
Since the unit test of the PJ I was in charge of was a hell picture, I will publish my own guide (?)