In-house test code culture starting with just one person

A story about a salaryman engineer who works for a company that does not have a test code culture, and works hard to spread the test culture while asking the manager's complexion. At the end of the article, I write a perspective on code review of tests.

Proposal to write test code in new development was rejected

"This is a release-first project. The idea of writing test code as a team in one word from the manager was rejected. I could understand why I shouldn't go on an adventure because I don't have the know-how to write test code in-house. However, it should be okay to write a test in my spare time, so I decided to do my best.

Test code culture starting with just one person

When I asked an engineer I knew about it, there were about two people who had written a test. I won't tell him, but he will be certified as a mentor and will teach you efficient methods and know-how in Python and Django. Looking back, I think I was blessed with the environment. There are multiple Python test frames, but the official documentation is solid Pytest. Pytest is highly functional and the documentation is complicated, so I think you can learn how to use it with Examples and use it as a dictionary.

Install pytest


pip install pytest

pytest test sample


# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals


#Function to test
def add(a, b):
    return a + b


#Test code function name is test_How to start with pytest
def test_add():
    assert add(1, 1) == 2
    assert add(1, 2) != 2

Execution result of pytest


>>> $ py.test ../tests/test_add.py 
=============================================================================== test session starts ===============================================================================
platform darwin -- Python 2.7.5 -- py-1.4.31 -- pytest-2.7.0
rootdir: /Users/***********, inifile: pytest.ini
plugins: cache, django, pep8, pythonpath
collected 2 items 

../tests/test_add.py ..

============================================================================ 2 passed in 0.06 seconds =============================================================================

Test coverage does not increase

At first, I worked hard to write unit tests, but the burden on developers is heavy. There is a feeling of trying too hard. When I consulted with him, he advised me to do a black-box test on a Web API basis instead of a function-based test on a social network server. When I tried it, the development man-hours were small, and it was simple and powerful. I still use it. Code coverage did not improve, but Web API testing can cover the API with less effort.

This test method is especially useful in scripting languages, where you will often find the kind of bugs that are detected at compile time.

Pytest code to test JSON-style Web API


# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
import ujson
import requests


def test_api():
    #Test the GitHub API
    url = "https://api.github.com/repos/vmg/redcarpet/issues?state=closed"
    headers = {'Accept-Encoding': 'identity, deflate, compress, gzip',
               'Accept': '*/*', 'User-Agent': 'python-requests/1.2.0',
               'Content-type': 'application/json; charset=utf-8',
               }
    response = requests.get(url, headers=headers)
    #HTTP Status code is 200
    assert response.status_code == 200

    #Being able to parse BODY with json
    data = ujson.loads(response.text)

    #Url in array, state, created_The existence of the element of at
    for param in data:
        assert 'url' in param
        assert 'state' in param
        assert 'created_at' in param

pytest execution result


>>> $ py.test ../tests/test_webapi.py 
=============================================================================== test session starts ===============================================================================
platform darwin -- Python 2.7.5 -- py-1.4.31 -- pytest-2.7.0
rootdir: /Users/*****, inifile: pytest.ini
plugins: cache, django, pep8, pythonpath
collected 2 items 

../tests/test_webapi.py ..
============================================================================ 2 passed in 0.87 seconds =============================================================================

Dye the development team into a test culture

When all the API tests of the server were completed, we made a rule to consult within the team and write the Web API test. Since pytest has a simple mechanism, anyone with sample code could easily learn it. However, it takes a long time to do muddy work such as pointing out in the review until it is fixed, executing tests regularly and fixing any errors, and telling new participants. (If you are familiar with it, run it regularly? I think you felt that it was an automatic test with commit hook. At that time, I was thinking of introducing it, but I forgot to introduce it without getting heavy. )

-Enabled to execute all tests with one command (written by shell) ・ In order to develop comfortably, all tests were executed within 60 seconds. ・ It was decided that the test would pass the work completion condition in the local environment construction procedure. ・ Confirmed that the test passed during code review ・ When developing a new function, it is a prerequisite that a test is attached at the time of code review. ・ I ran a test on a regular basis and had them fix it or fix it.

Prayer deploy and bugs have been reduced in the operational stage!

The app was successfully released and a test culture was taking root within the team. Development is done on a pull request basis, with test code attached without pointing out. Due to the test failure, we found a dependency between the newly developed module and the module that was thought to be out of scope, and we were able to prevent production bugs.

As an operation rule, the checklist at the time of deploy always clearly states that test execution is performed, so the habit of executing tests from the development stage has taken root among the members.

This time spread the test culture within the company

I held an in-house study session and told him that it would be good to write a test. The reason why our project was successful (one of them) was that I wrote test code. Especially in the company, it was effective because I could share the actual test code.

Tests that never end, tests that do not run

It's a test of time, and if you continue to operate it for a long time, the test code will carry technical debt. A problem child test that needs to be corrected every time you make a change, a test that takes more than 300 seconds to execute once, and a test code that is left commented out because an error occurred due to a problem with the test code itself when deploying in a hurry. Test code that is too complicated to fix and you don't know what you are testing.

A declining test culture within the team

It has been one year since the operation started. Tests that initially completed in less than 60 seconds now take 12 minutes to complete. The rule to run the test on deploy is also omitted due to latency. Many of the engineers were veterans and the debugging team was excellent, so the bugs did not increase. Many people wrote tests for the new module pull request, but the test culture within the project has declined.

The seeds of test culture are growing

Due to the influence of the in-house study session and the fact that some engineers had already written the test code individually, although I intended to start it alone, the number of new projects that gradually adopt the test writing as a rule is increasing. went. I'm doing a test! I think it is important to say that.

end




Looking back: Where did you go wrong, did your team run out?

If you neglect to implement the business logic, it will not work according to the specifications, so it is essential to implement it, but it is quite good even if you do not implement or execute the test code. I think the reason why the test culture has declined is that the development team is exhausted from the maintenance of the test, so I will write down where it was exhausted.

1-1. Test execution by hand or test execution in the local environment is not good

I think we should have linked commit hook and jenkins, or adopted Travis CI or Circle CI and automatically tested at the timing of commit. I wrote about how to use CircleCI in Django in this article

1-2. Poor quality tests should have been thrown away

The team was exhausted by the maintenance of the test, which is complicatedly dependent on master data and fails once every two times due to the problem of the test itself. I think that poor quality tests should have been thrown away and rewritten for metabolism.

1-3. Long-running tests are omitted

I can't wait for a test that takes more than 60 seconds. Since the tempo of development will be slow, I think that we should have considered more consideration so that we can develop comfortably by considering parallel execution of tests and dividing it into two layers such as quick test and full test.

1-4. I should have left a comment on what perspective I was testing

Although an error will occur if it is operated for a long period of time, the number of variables are lined up and the person who wrote it does not know what test it is. I think it was.

2. Know-how for writing test code as a team

2-1. Describe that the test ends normally in the completion condition of the environment construction procedure.

This is an effective measure for new participants. Even after construction, if you check whether the test passed for each pull request, it will be fixed quickly.

2-2. Pull request development is effective for team writing test code

If you proceed with development based on pull request, control will be effective, so you can prevent the situation where the number of untested code increases, and the educational effect is great.

2-3. Don't leave a broken test

When a test failure occurs, no one will see the test result, so we will give priority to fixing it. If someone fixes it on a full-time basis, it will not improve forever, so it may be good for education if you carry it around on duty.

2-4. Pay attention to the test execution time. Slow test is bad

If the test execution time is slow, you will not be able to see the results, so you should take steps to execute the tests in parallel or speed up.

3. Perspectives on code review of tests

This is the point of view to check in the code review when receiving a pull request.

3-1. Does the test pass?

Automate or actually check out to make sure the test passes.

3-2. Is the test execution time appropriate?

Unless you have a specific reason, one test should be completed within 5 seconds. Check for any tests that take a long time in terms of execution time.

3-3. Is it independent of master data or is there a fixed value?

Fixed values such as card_id = 1 are technical debt that will result in an error if the ID is changed in the future. Insist that you load the test value in fixtures instead of a fixed value, or pull the ID from another data acquisition API.

3-4. Are there any comments about appropriate test items?

Check in terms of what perspective the test is described in. After a long period of operation, the members will be replaced, so even if an error occurs in the test, the reason will not be clear.

3-5. Is there a dependency due to the execution order?

If you write a separate test, for example, register and then delete, a dependency will occur. If there is a dependency on the execution order of the tests, an error will occur when parallelizing the tests for speed, so if there is a dependency, point out that it should be combined into one.

Recommended Posts

In-house test code culture starting with just one person
Strengthen with code test ⑦
Strengthen with code test ⑨
Strengthen with code test ③
Strengthen with code test ⑤
Strengthen with code test ②
Strengthen with code test ①
Strengthen with code test ⑧
Strengthen with code test ⑨
Test automation starting with L-Chika (3) Oscilloscope integration
How to start the code written in Atom with one command without starting teminal
Airtest, one question at a time. Unity app E2E test starting with Airtest and Poco