A story about trying to improve the testing process of a system written in C language for 20 years

Introduction

A while ago, I tried to improve various tests of a system written in C language for 20 years, so I will write a little knowledge I got at this time.

  • Caution Since I am creating an environment on my personal computer while remembering those days to write articles, it is actually different from the environment and version I did in practice. Also, there is some code in this article, but it's all just a fake example I thought of to write the article.

Creating test code for unit tests

With a system that has been running for 20 years, no one knows what it means anymore, but there are a number of places where existing behavior should not be changed.

If you have no choice but to work on such a part, the following methods are effective.

First, write the test code for the existing code. And after confirming that all pass, we will expand the function little by little. This makes it possible to proceed with the function addition work while confirming that the new function addition does not break the existing function.

It's a so-called ** test first **, but in such a conservative environment with a long history, if you do what you said in XP, various distortions will occur. We'll talk about that at the end of this chapter.

Introducing xUnit

You can do your best with assert etc., but it is better to use xUnit test framework such as CUnit.

** Cutter ** (1.2.7 released on 2019-09-13 is the latest) https://cutter.osdn.jp/index.html.ja

** UNITY ** (v2.4.3 released on 2017/11. Some commit history is 2019/10) http://www.throwtheswitch.org/unity https://github.com/ThrowTheSwitch/Unity

You can also test from xUnit frameworks for C ++ such as CPPUnit, but this time I removed it.

Why xUnit test framework is needed

It is better to use a well-known test framework because there are many examples of realization in the world and it will be helpful in case of trouble. Also, using the well-known test framework makes it easier to work with Jenkins.

For example, Jenkins's xUnit Plugin has a function to aggregate XML output by famous xUnit, and it is easy to automatically test. It is possible to aggregate the results of.

CUnit CUnit is an xUnit framework for C language that can be used from gcc or visual c.

How to build with visual studio 2019

Since there are only solutions from the VS2008 and 2005 eras, it is old and warnings come out.

  1. Open CUnit-2.1-3 / VC9 / CUnit.sln
  2. Create CUnit.h based on CUnit-2.1-3 / CUnit / Headers / CUnit.h.in. The differences are as follows. image.png
  3. Build in the order of libcunit-> other projects.

** How to check the number of simultaneous builds of a project in Visual Stduio 2019 ** Select "Tools"-> "Options" from the menu to open the options dialog, and select "Projects and Solutions"-> "Build / Run" to display the corresponding screen. image.png

How to build with make command

The following is an example of building using the make command under the environment of Windows 10 + Ubuntu 18.04. You can basically build it in the same way as the following article, but the version is 2.1-3.

** Unit test C program with CUnit ** https://qiita.com/muniere/items/ff9a984ed7e51eee7112

wget https://jaist.dl.sourceforge.net/project/cunit/CUnit/2.1-3/CUnit-2.1-3.tar.bz2
tar xvf CUnit-2.1-3.tar.bz2
cd CUnit-2.1-3
aclocal
autoconf
automake
./configure
make
make install
troubleshoot
No automake or autoconf command

Execute the following to install.

sudo apt-get install autoconf              
When I run configure or something, I get the error "configure: error: cannot find install-sh, install.sh, or shtool in". "" ./.. "" ./../ .. "".

Execute the following command and start over from autoconf

autoreconf -i
"Error: Libtool library used but'LIBTOOL' is undefined" occurred during configure

Install libtool with the following command.

sudo apt-get install libtool
If "error while loading shared libraries: libcunit.so.1: cannot open shared object file: No such file or directory" appears at runtime

Set the environment variable LD_LIBRARY_PATH.

export LD_LIBRARY_PATH=/usr/local/lib

If you want to make it persistent, write it in .bashrc as well.

Execution example

When executing from the console, it will be as follows.

#include <CUnit/CUnit.h>

int max(int x, int y) {
	if (x>y) {
		return x;
	} else {
		return y;
	}
}
int min(int x, int y) {
	/** bug  **/
	if (x>y) {
		return x;
	} else {
		return y;
	}
}

void test_max_001(void) {
	CU_ASSERT_EQUAL(max(5, 4) , 5);
}
void test_max_002(void) {
	CU_ASSERT_EQUAL(max(4, 5) , 5);
}
void test_max_003(void) {
	CU_ASSERT_EQUAL(max(5, 5) , 5);
}
void test_min_001(void) {
	CU_ASSERT_EQUAL(min(5, 4) , 4);
}
void test_min_002(void) {
	CU_ASSERT_EQUAL(min(4, 5) , 4);
}
void test_min_003(void) {
	CU_ASSERT_EQUAL(min(5, 5) , 5);
}

int main() {
	CU_pSuite max_suite, min_suite;

	CU_initialize_registry();
	max_suite = CU_add_suite("max", NULL, NULL);
	CU_add_test(max_suite, "test_001", test_max_001);
	CU_add_test(max_suite, "test_002", test_max_002);
	CU_add_test(max_suite, "test_003", test_max_003);

	min_suite = CU_add_suite("min", NULL, NULL);
	CU_add_test(min_suite, "test_001", test_min_001);
	CU_add_test(min_suite, "test_002", test_min_002);
	CU_add_test(min_suite, "test_003", test_min_003);
	CU_console_run_tests();
	CU_cleanup_registry();

	return(0);
}

See below for other functions for ASSERT. http://cunit.sourceforge.net/doc/writing_tests.html

Please link to libcunit.a when compiling the source that runs CUnit.

gcc unit.c  -Wall -L/usr/local/lib -lcunit -o unit

When executed from the console, it will be as follows. image.png

How to work with Jenkins

Learn how to get CUnit results into Jenkins.

Make the test code output XML

Modify the code to create an XML file to pass to Jenkins. When outputting to an XML file, use CU_automated_run_tests and CU_list_tests_to_file instead of CU_console_run_tests.

/**Abbreviation**/
	/**Console output
	CU_console_run_tests();
	*/
	/**XML output**/
	CU_set_output_filename("./unit");
	CU_automated_run_tests();
	CU_list_tests_to_file();
	CU_cleanup_registry();
/**Abbreviation**/

If you change the above file, compile and execute it, the following XML file will be created instead of outputting to the console.

See below for details on how to run CUnit. http://cunit.sourceforge.net/doc/running_tests.html#overview

Make Jenkins parse the test result XML.

Install xUnit Plugin in Plugin Management. image.png

In the job settings, add "Publish xUnit test result report" to the post-build process and select CUnit-2.1.

image.png

Specify unit-Results.xml for Pattern.

When you run the build, you will get the following result: image.png

image.png

Creating test stubs

When automating tests, if you prepare only the xUnit test framework ** and try to automate it, the introduction will often fail. Either you can only test simple utility functions and they're obsolete, or you end up writing complex test code to make the dependent libraries consistent.

For example, suppose you have library 2 that depends on library 1 as shown below. image.png

When testing library 2, create a stub for testing library 1 so that it returns convenient data for testing library 2. That way, even if library 1 depends on the network and database, you can put it in a convenient state for testing without actually using them. image.png

Example of creating a simple stub

Consider how to write a test for library 2 that depends on library 1 with a simple example.

Library 1 sample

Library 1 is implemented as follows. Prepare a function called add that just takes two arguments and returns the result of adding them.

static1.h


#ifndef STATIC1_H
#define STATIC1_H
extern int add(int x, int y);
#endif

static1.c


#include "static1.h"

int add(int x, int y) {
	return x + y;
}
Library 2 sample

Library 2 has the following implementation. In library 2, the proc function uses the add function of library 1 for processing.

static2.h


#ifndef STATIC2_H
#define STATIC2_H

#include "static1.h"

extern int proc(int x, int y, int z);

#endif

static2.h


#include "static2.h"

int proc(int x, int y, int z) {
	int a = add(x,y);
	return add(a, z);
}
Library 1 stub

Create stubs instead of real code. The processing of this stub performs the following processing each time the add function is executed. ・ Record the number of calls -Execution of the callback function registered in advance in the test code

stub_static.h


#ifndef STUB_static1_H
#define STUB_static1_H
#include "static1.h"

typedef int (*pfunc_add_def) ( int x , int y );

typedef struct {

    int count_add;
    pfunc_add_def pfunc_add;
} stub_data_def_static1;
extern stub_data_def_static1 stub_data_static1;
#endif

stub_static.c


#include "stub_static1.h"
stub_data_def_static1 stub_data_static1;


int add ( int x , int y ) {
    ++stub_data_static1.count_add;
    
    return stub_data_static1.pfunc_add( x , y );
    
}

Test code example

The following is an example of executing the proc function by defining the callback function executed from the stub of the add function in the test code.

#include <CUnit/CUnit.h>
#include "static2.h"
#include "stub_static1.h"

int test001_stub_add(int x, int y) {
	/** add()Stub. X and y for testing with CUNIT**/
	if (stub_data_static1.count_add == 1) {
		/**First call**/
		CU_ASSERT_EQUAL(x , 10);
		CU_ASSERT_EQUAL(y , 5);
		return 15;
	}
	if (stub_data_static1.count_add == 2) {
		CU_ASSERT_EQUAL(x , 15);
		CU_ASSERT_EQUAL(y , 4);
		return 19;
	}
	CU_FAIL("Incorrect call count of add");
	return -1;
}

void test001() {
	memset(&stub_data_static1, 0, sizeof(stub_data_static1));
	stub_data_static1.pfunc_add = test001_stub_add;
	int ret = proc(10, 5,4);
	
	/**Check the number of calls**/
	CU_ASSERT_EQUAL(stub_data_static1.count_add , 2);
	
	CU_ASSERT_EQUAL(ret , 19);
}


int main() {
	CU_pSuite suite;

	CU_initialize_registry();
	suite = CU_add_suite("stubsample", NULL, NULL);
	CU_add_test(suite, "test001", test001);
	CU_console_run_tests();
	CU_cleanup_registry();

	return(0);
}

If the stub function is called multiple times, it checks the value of the argument that the stub will receive based on the number of calls and determines the value that the stub should return.

Automatic stub creation

In the previous section, we introduced an example of a simple test code using a stub. The next issue is the cost of creating stubs.

It is not impossible to create it by hand, but if there are a large number of functions in library 1 or if they are updated frequently, it can be difficult to modify them by hand. So let's think about how to automatically generate stubs.

It's basically a simple conversion. Therefore, any programming language can automatically generate stubs, but if you want to have fun, a programming language that can use the template library and C language parser is recommended.

Try to automatically generate stubs in Python

C language parser library

You can analyze C / C ++ source code by using pycparser. https://github.com/eliben/pycparser

Note that this library only works for code that has been processed by the preprocessor. Therefore, in fact, use the result of preprocessor execution as shown below for the source code to be analyzed.

gcc -E static1.h

This time, we will parse the following precompiled header files.

Header file to be analyzed


extern int add(int x, int y);
typedef struct {
 int x;
 int y;
} in_data_t;
typedef struct {
 int add;
 int minus;
} out_data_t;
extern void calc(in_data_t* i, out_data_t *o);
extern int max(int data[], int length);

Sample to parse header file


import sys
from pycparser import c_parser, c_ast, parse_file

# https://github.com/eliben/pycparser/blob/master/examples/cdecl.py
def _explain_type(decl):
    """ Recursively explains a type decl node
    """
    typ = type(decl)

    if typ == c_ast.TypeDecl:
        quals = ' '.join(decl.quals) + ' ' if decl.quals else ''
        return quals + _explain_type(decl.type)
    elif typ == c_ast.Typename or typ == c_ast.Decl:
        return _explain_type(decl.type)
    elif typ == c_ast.IdentifierType:
        return ' '.join(decl.names)
    elif typ == c_ast.PtrDecl:
        quals = ' '.join(decl.quals) + ' ' if decl.quals else ''
        return quals + _explain_type(decl.type) + '*'
    elif typ == c_ast.ArrayDecl:
        arr = 'array'
        if decl.dim: 
            arr = '[%s]' % decl.dim.value
        else:
            arr = '[]'

        return _explain_type(decl.type) + arr

    elif typ == c_ast.FuncDecl:
        if decl.args:
            params = [_explain_type(param) for param in decl.args.params]
            args = ', '.join(params)
        else:
            args = ''

        return ('function(%s) returning ' % (args) +
                _explain_type(decl.type))

    elif typ == c_ast.Struct:
        decls = [_explain_decl_node(mem_decl) for mem_decl in decl.decls]
        members = ', '.join(decls)

        return ('struct%s ' % (' ' + decl.name if decl.name else '') +
                ('containing {%s}' % members if members else ''))

def show_func_defs(filename):
    # Note that cpp is used. Provide a path to your own cpp or
    # make sure one exists in PATH.
    ast = parse_file(filename, use_cpp=False,
                     cpp_args=r'-Iutils/fake_libc_include')
    if not isinstance(ast, c_ast.FileAST):
        return

    for ext in ast.ext:
        if type(ext.type) == c_ast.FuncDecl:
            print(f"function name: {ext.name} -----------------------")
            args = ''
            print("parameters----------------------------------------")
            if ext.type.args:
                for arg in ext.type.args:
                    print(f"{_explain_type(arg)} {arg.name}")
            print("return----------------------------------------")
            print(_explain_type(ext.type.type))

if __name__ == "__main__":
    if len(sys.argv) > 1:
        filename  = sys.argv[1]
    else:
        exit(-1)

    show_func_defs(filename)

As a simple flow of the analysis process, FileAST can be obtained as the execution result of parse_file. By parsing the ext property there, you can get the information of the function defined in the file. The result of executing the above program and checking the function definition in the header file is output as follows.

C:\dev\python3\cparse>python test.py static1.h
function name: add -----------------------
parameters----------------------------------------
int x
int y
return----------------------------------------
int
function name: calc -----------------------
parameters----------------------------------------
in_data_t* i
out_data_t* o
return----------------------------------------
void
function name: max -----------------------
parameters----------------------------------------
int[] data
int length
return----------------------------------------
int

It was confirmed that C language source files can be easily analyzed by using pycparser in this way.

Template library

When creating header and source files for stubs, the template library makes it easier to create. Please refer to the following for the template library in Python.

** I want to output HTML in Python for the first time in a while, so check the template ** https://qiita.com/mima_ita/items/5405109b3b9e2db42332

This time I am using Jinja2.

Code for creating stubs

The following is an example of implementing a stub creation tool using a C language parser and template library.

create_stub.py


import sys
import os
from pycparser import c_parser, c_ast, parse_file
from jinja2 import Template, Environment, FileSystemLoader


stub_header_tpl = """
#ifndef STUB_{{name}}_H
#define STUB_{{name}}_H
#include "{{name}}.h"
{% for func in function_list %}
typedef {{func.return_type}} (*pfunc_{{func.name}}_def) ({% for arg in func.args %}{% if loop.index != 1 %},{% endif %} {{arg.typedef}} {{arg.name}}{{arg.array}} {% endfor %});
{% endfor %}

typedef struct {
{% for func in function_list %}
    int count_{{func.name}};
    pfunc_{{func.name}}_def pfunc_{{func.name}};
{% endfor %}
} stub_data_def_{{name}};
extern stub_data_def_{{name}} stub_data_{{name}};
#endif
"""

stub_source_tpl = """
#include "stub_{{name}}.h"
stub_data_def_{{name}} stub_data_{{name}};

{% for func in function_list %}
{{func.return_type}} {{func.name}} ({% for arg in func.args %}{% if loop.index != 1 %},{% endif %} {{arg.typedef}} {{arg.name}}{{arg.array}} {% endfor %}) {
    ++stub_data_{{name}}.count_{{func.name}};
    {% if func.return_type != "void" %}
    return stub_data_{{name}}.pfunc_{{func.name}}({% for arg in func.args %}{% if loop.index != 1 %},{% endif %} {{arg.name}} {% endfor %});
    {% else %}
    stub_data_{{name}}.pfunc_{{func.name}}({% for arg in func.args %}{% if loop.index != 1 %},{% endif %} {{arg.name}} {% endfor %});
    {% endif %}
}
{% endfor %}
"""


class ParameterData():
    def __init__(self, name, typedef):
        self.__name = name
        self.__typedef = typedef
        self.__array = ""
        if '[' in typedef:
            self.__array = "[" + typedef[typedef.index("[")+1:typedef.rindex("]")] + "]"
            self.__typedef = typedef[0: typedef.index("[")]

    @property
    def name(self):
        return self.__name

    @property
    def typedef(self):
        return self.__typedef

    @property
    def array(self):
        return self.__array

class FunctionData:
    def __init__(self, name, return_type):
        self.__name = name
        self.__return_type = return_type
        self.__args = []

    @property
    def name(self):
        return self.__name

    @property
    def return_type(self):
        return self.__return_type

    @property
    def args(self):
        return self.__args


# https://github.com/eliben/pycparser/blob/master/examples/cdecl.py
def _explain_type(decl):
    """ Recursively explains a type decl node
    """
    typ = type(decl)

    if typ == c_ast.TypeDecl:
        quals = ' '.join(decl.quals) + ' ' if decl.quals else ''
        return quals + _explain_type(decl.type)
    elif typ == c_ast.Typename or typ == c_ast.Decl:
        return _explain_type(decl.type)
    elif typ == c_ast.IdentifierType:
        return ' '.join(decl.names)
    elif typ == c_ast.PtrDecl:
        quals = ' '.join(decl.quals) + ' ' if decl.quals else ''
        return quals + _explain_type(decl.type) + '*'
    elif typ == c_ast.ArrayDecl:
        arr = 'array'
        if decl.dim: 
            arr = '[%s]' % decl.dim.value
        else:
            arr = '[]'

        return _explain_type(decl.type) + arr

    elif typ == c_ast.FuncDecl:
        if decl.args:
            params = [_explain_type(param) for param in decl.args.params]
            args = ', '.join(params)
        else:
            args = ''

        return ('function(%s) returning ' % (args) +
                _explain_type(decl.type))

    elif typ == c_ast.Struct:
        decls = [_explain_decl_node(mem_decl) for mem_decl in decl.decls]
        members = ', '.join(decls)

        return ('struct%s ' % (' ' + decl.name if decl.name else '') +
                ('containing {%s}' % members if members else ''))

def analyze_func(filename):
    ast = parse_file(filename, use_cpp=False,
                     cpp_args=r'-Iutils/fake_libc_include')
    if not isinstance(ast, c_ast.FileAST):
        return []
    function_list = []
    for ext in ast.ext:
        if type(ext.type) != c_ast.FuncDecl:
            continue
        func = FunctionData(ext.name,  _explain_type(ext.type.type))
        if ext.type.args:
            for arg in ext.type.args:
                param = ParameterData(arg.name, _explain_type(arg))
                func.args.append(param)
        function_list.append(func)
    return function_list


if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("python create_stub.py preprocessed header output folder")
        exit(-1)
    filename  = sys.argv[1]
    dst_folder  = sys.argv[2]
    function_list = analyze_func(filename)
    if len(function_list) == 0:
        print("Could not find function")
        exit(-1)
    data = {
        'name' : os.path.splitext(filename)[0],
        'function_list' : function_list
    }
    #Creating a stub header file
    with open(f"{dst_folder}/stub_{data['name']}.h", mode='w', encoding='utf8') as f:
        f.write(Template(stub_header_tpl).render(data))

    #Creating a stub source file
    with open(f"{dst_folder}/stub_{data['name']}.c", mode='w', encoding='utf8') as f:
        f.write(Template(stub_source_tpl).render(data))

The stubs created using this tool are as follows.

stub_static1.h



#ifndef STUB_static1_H
#define STUB_static1_H
#include "static1.h"

typedef int (*pfunc_add_def) ( int x , int y );

typedef void (*pfunc_calc_def) ( in_data_t* i , out_data_t* o );

typedef int (*pfunc_max_def) ( int data[] , int length );


typedef struct {

    int count_add;
    pfunc_add_def pfunc_add;

    int count_calc;
    pfunc_calc_def pfunc_calc;

    int count_max;
    pfunc_max_def pfunc_max;

} stub_data_def_static1;
extern stub_data_def_static1 stub_data_static1;
#endif

stub_static.c



#include "stub_static1.h"
stub_data_def_static1 stub_data_static1;


int add ( int x , int y ) {
    ++stub_data_static1.count_add;
    
    return stub_data_static1.pfunc_add( x , y );
    
}

void calc ( in_data_t* i , out_data_t* o ) {
    ++stub_data_static1.count_calc;
    
    stub_data_static1.pfunc_calc( i , o );
    
}

int max ( int data[] , int length ) {
    ++stub_data_static1.count_max;
    
    return stub_data_static1.pfunc_max( data , length );
    
}

When the C language parser is not available

There are often situations where Python is not available or the C parser is not available. In that case, use doxygen.

Doxygen can parse the source file and output the function information to XML. It is also possible to create a file for stubs based on this XML. Many C and C ++ sites use doxygen, so it's probably a lower barrier to adoption than Python. *

How to create other stubs and mock

I didn't actually use it, but there are other methods as follows.

CMock https://github.com/ThrowTheSwitch/CMock It seems to create code for mock using Ruby. It may be possible if Ruby can be used in the environment.

** Ideas for using stub functions in C testing ** https://qiita.com/HAMADA_Hiroshi/items/e1dd3257573ea466d169 "How to perform unit tests by stubing functions in the same source without modifying the existing source", basic idea Uses macros to replace existing functions with aliases. Fortunately, I didn't have to write a test to call a function in the same source, so I didn't actually use it.

How to measure coverage rate

When you write test code, you can judge whether you are executing the test without exception by measuring the ratio covered by the test code. Fortunately, gcc comes with a tool called gcov to measure coverage.

10 gcov—a Test Coverage Program https://gcc.gnu.org/onlinedocs/gcc/Gcov.html

** How to use gcov ** https://mametter.hatenablog.com/entry/20090721/p1

Coverage measurement example

Let's measure the coverage with the example of the stub used earlier. This time, in order to intentionally create an unreachable route, modify static2.c of library 2 as follows.

static2.c


#include "static2.h"

int proc(int x, int y, int z) {
	int a = add(x,y);
	return add(a, z);
}

int proc2() {
	/**Not reached**/
	return 0;
}

Then compile with the -coverage option to measure coverage.

gcc -c -Wall -Wextra static2.c -coverage
ar r libstatic2.a static2.o 
gcc -c -Wall -Wextra stub_static1.c
ar r libstub_static1.a stub_static1.o
gcc test.c -coverage  -Wall -Wextra -L. -lstatic2 -lstub_static1   -L/usr/local/lib -lcunit -o test

Executing the above command will create the following files.

After that, running test will create the following files.

To check the coverage of static2.c under test, execute the following command.

gcov static2.gcda 

When this command is executed, the following message will be displayed.

File 'static2.c'
Lines executed:60.00% of 5
Creating 'static2.c.gcov'

Here you can see that the coverage rate is 60%, and to find out where it is not covered, display static2.c.gcov. In this example, you can see that proc2 is not passing. image.png

Other automated tests

About the effect

We judge that it is effective in the following points.

・ The risk of expanding the function without changing the behavior of the current function can be suppressed. -By writing a test code, you can clarify the functions that were in the dark until now. ・ If you write test code, you can detect defects that have been sleeping in existing code. → Whether or not to fix it is another matter. A balance between the frequency of occurrence, the degree of impact at the time of occurrence, and the correction cost.

However, there are some issues, so let's consider them in the next section.

What about SQL exams?

In a legacy environment, you may have to write SQL in the source code and actually connect it to a database and run it for testing. In this case, it is desirable to divide the DB area used by each developer and perform unit tests.

However, in a large organization, there are departments that specialize in DB, and it is not easy to tamper with the database. Also, if you put the database itself in the local environment, a poor PC becomes a bottleneck, or even if you break through it, if you leave the database management in your personal environment, some developers will continue to use the old environment. There is also a risk of doing so.

In such a case, you can handle it by a less conscious method of rolling back when the automatic test is over.

① Start automatic test ② Start a DB transaction ③ Store test data in DB ④ Perform an automatic test ⑤ Check how the DB has changed based on the results of the automatic test. ⑥ DB rollback

This method is not a perfect idea, but it is easy to implement.

Massive combination testing

If you write the input value and expected value in an array of structures, you can do a lot of tests. A common method is to write a list of inputs and outputs in Excel, convert it to an array of C language structures with a VBA macro, and write it in the test code.

Distortion doing test first in a conservative waterfall environment

If this method is used in a conservative waterfall environment, after implementation, you will create test specifications and checklists for unit tests and perform tests. However, if you do test-first things, the implementation will be completed = the unit test will be completed, so distortion will occur here.

Plans don't get on the traditional line

With the conventional method, the schedule line is described in three steps: implementation, creation of a checklist for unit tests, and implementation of unit tests. And with a highly controlled, complete and perfect organization, each process is timed exactly.

If you do test-first things, unit tests cannot be separated from the implementation, so individual work time cannot be measured, and you will report them separately.

Unit test specifications

You need to decide what to do with unit test specifications and checklists that were created in the traditional way. It would be nice if we could decide that the code review of the test code would be OK and that the product of the unit test would be the test code, but in a legacy environment, documents are important. In this case, you need to create test specifications and checklists from the test code.

The first countermeasure is to manually post the contents of the test code to the checklist. The problem is that the test code and test specifications get out of sync. Also, this method is a very bad response that eliminates the advantage of running tests automatically, but it is often adopted because anyone can do it if it takes time.

The second method is to generate a test specification from the comments in the test code. Describe the content of the test in the comment of the test code and summarize the results extracted by Doxygen etc. in the checklist. In this case, there is a risk that there will be a discrepancy between the comment and the actual processing.

The third method is to write Japanese that does not feel strange even if it is output to the checklist for the test name specified in the test code.

	CU_add_test(max_suite, "max(a,b)To run a=5,b=In case of 4, confirm that the value of a is obtained", test_max_001);

If you execute the test with the above description, the following will be output to the XML of the test result. image.png

After that, output the XML checklist and you're done. The problem with this method is that the check contents performed by CU_ASSERT_EQUAL etc. are not output to XML, so the granularity of the checklist becomes large.

The fourth method is to create a function that wraps CU_ASSERT_EQUAL etc., and within that function, output the inspection contents to a file in a form that is easy to output to the checklist.

Regardless of which method you choose, be aware that sticking to the document can be a hassle. → In a culture that sticks to writing, let's make sure that such work is included in the estimate.

Handling of failure slips in unit tests

Since the unit tests are finished when the implementation is finished, it is impossible to raise the failure in the unit tests. However, a highly controlled, complete and perfect organization may focus on unit test failures. In this case, the problem that occurred during implementation will be appropriately made up as a bug like that.

The theory that should be supposed to be is to persuade and create new rules, but I think it is impossible because the culture is different.

The digits of the number of tests change, so conventional indicators become useless

Introducing automated testing makes it possible to mechanically process a large number of combinatorial tests. As a result, the number of tests performed automatically is larger than the number of tests performed by the conventional method. (That's an order of magnitude)

What used to be at most hundreds of checklist items is now usually a thousand or a million. I'm talking about "what's wrong with doing a lot of tests?", But if it is a highly controlled, complete and perfect organization, it will be treated as an abnormal value, so adjust the particle size appropriately and use it for reporting. It may be necessary to adjust the number to that level.

In addition, if you want to take proper measures, it would be a good idea to create a new index as a separate tabulation for automated testing.

Since the number of failures after the integration test is less than before, it is treated as an abnormal value.

If you do a comprehensive unit test, you will not have any problems with the integration test.

This is a problem with this. For example, let's say you've had dozens of failures in the past, but the number of defects is almost gone. If you're only looking at the numbers, you can't help wondering if the integration test perspective is wrong.

Many people don't want to write test code

As I wrote below, that's what it is. Let's give up.

** Automation of unit tests using unconscious Visual Studio ** https://qiita.com/mima_ita/items/05ce44c3eb1fd6e9dd46

It is possible to explain the effect of the automatic test by man-hours and quality, convert it into a monetary amount and raise a sense of crisis, and prepare educational materials, but honestly, it is a culture that can not be understood even if such a story is accumulated Does not work ** at all ** </ font>.

However, in any organization, there are only a few people with aspirations, so let's expect such people.

Static analysis

We talked about automated testing in the previous section. After all, this is just actually moving and checking the range where you wrote the test code. This section describes how to find a dubious implementation without actually writing and running test code.

Find suspicious parts with static analysis tools

By using a static analysis tool, it is possible to detect a part with a dubious implementation. Expensive tools can make effective discoveries, but open source is also effective enough.

For C language, you can use CppCheck. http://cppcheck.sourceforge.net/

cppcheck can be run from Windows with GUI or linux based on CUI.

How to use from the command line

I will explain how to use cppcheck under the environment of Windows10 + Ubuntu 18.04.

First, install it with the following command.

sudo apt-get install cppcheck

After the installation is complete, cppcheck will be executed by executing the following command.

$ cppcheck --enable=all .
Checking cov.c ...
1/7 files checked 14% done
Checking main.c ...
2/7 files checked 28% done
Checking static1.c ...
3/7 files checked 42% done
Checking static2.c ...
4/7 files checked 57% done
Checking stub_static1.c ...
5/7 files checked 71% done
Checking test.c ...
6/7 files checked 85% done
Checking unit.c ...
7/7 files checked 100% done
[cov.c:7]: (style) The function 'bar' is never used.
[static1.c:7]: (style) The function 'calc' is never used.
[static2.c:8]: (style) The function 'proc2' is never used.
(information) Cppcheck cannot find all the include files (use --check-config for details)

Try using cppcheck with Windows GUI

(1) Please download the Installer from the following page. http://cppcheck.sourceforge.net/

(2) After installation, start CppCheck and the following screen will open. image.png

(3) Select "File"-> "New Project" from the menu. image.png

(4) Set an arbitrary path to save the project file and click "Save". image.png

(5) The "Project File" dialog opens. Click the "Add" button to add the path containing the C language source file. image.png

image.png

image.png

(6) Click OK in the "Project File" dialog, and you will be asked if you want to create a build directory. Select "Yes". image.png

(7) C files under the specified directory are listed. image.png

(8) Expand the tree structure of the listed files to see the warning details. image.png

Collaboration with Jenkins

By using Cppcheck Plulgin of Jenkins, it is possible to aggregate the results of CppCheck in Jenkins. https://plugins.jenkins.io/cppcheck

(1) Select and install CppCheck in the Jenkins plugin manager. image.png

(2) Execute cppcheck as follows and get the result in xml.

cppcheck --enable=all --xml --xml-version=2 ~/libsample/. 2>cppcheck.xml

(3) Add Publsh Cppcheck results to the post-build process. image.png

image.png

(4) When you execute the build, the CppCheck item will be displayed in the result. image.png

image.png

Measure metrics

The number of steps in the source code and the cyclomatic complexity are one of the indicators for predicting the quality of the source code.

** Cyclomatic complexity ** https://ja.wikipedia.org/wiki/%E5%BE%AA%E7%92%B0%E7%9A%84%E8%A4%87%E9%9B%91%E5%BA%A6

To put it simply, the more loops and branches, the higher the complexity. The higher the complexity, the more likely it is to contain bugs, so this metric can be used as a starting point for where to focus your testing.

Source Monitor Source Monitor is a source code metric written in C ++, C, C #, VB.NET, Java, Delphi, Visual Basic (VB6). It is a free software that can measure.

(1) Download it from the following page and extract it to any folder. http://www.campwoodsw.com/sourcemonitor.html

(2) When the expanded SourceMonitor.exe is executed, the following screen will be displayed. Select "Start Source Monitor". image.png

(3) Select [File]-> [New Project] from the menu to create a project that aggregates metrics. image.png

(4) The "Select Language" dialog opens. Select "C" and click "Next". image.png

(5) Specify the file name of the project file and the folder to store it, and click "Next". image.png

(6) Specify the folder containing the source code to be analyzed and click "Next". image.png

(7) Select an option and click "Next". image.png

option Description
Use Modified Complexity Metrix The default complexity metric is calculated by counting the cases in each switch statement, but by turning on the check, each switch block will be counted only once, and each case statement will be modified. It will no longer contribute to Complexity.
Do not count blank line Ignore blank lines when counting lines.
Ignore Continuous Header and Footer Comments Check this if you want to ignore consecutive comment lines that contain information such as file history automatically generated by version control utilities such as CVS and Visual SourceSafe.

(8) Click "Next". image.png

(9) If UTF8 files are included, check "Allow parsing of UTF-8 files" and click "Next". image.png

(10) Click "Finish". image.png

(11) A list of files to be measured will be displayed. After confirming, click OK. image.png

(12) The measurement result is displayed. image.png

Click the list to see a list of files. image.png image.png

Click the file item in the list to see the details. image.png

You can check the contents of the file by right-clicking the item of the file in the list to display the context menu and selecting "View Source File". image.png image.png

Frequently used numbers

name Description
Lines The number of physical lines in the source file. If "Do not count blank line" is checked in the project creation, blank lines are not included. Used to see a rough scale
Max Complexity Cyclomatic complexity. The handling of the case in the switch clause changes depending on whether or not "Use Modified Complexity Metrix" is checked when creating the project. The higher this number, the harder it is to test.
Max Depth Maximum nesting depth. Keep an eye out for deeply nested code as it tends to be complicated
% Comment Percentage of comments. 0%On the contrary, if it is too big, you should check it.

Find similar code

It is possible to detect the copied code by using PMD-CPD introduced below.

** Do you want me to fix that copy and paste? ** https://qiita.com/mima_ita/items/fe1b3443a363e5d90a21

Unlike at the time of development, it is not used for the purpose of finding similar code and making it common. ** Unlike under development, a system that is already in operation cannot be easily refactored, such as standardization, even if the code is copied and pasted. </ font> **

The purpose of detecting code clones in these situations is to find out if there are other similarities that need to be modified, for example, when a failure response or function addition occurs and one source code is modified.

For example, if you make a mistake, you will naturally be asked, "Is another part okay?" At this time, you can cheat by saying something like "I used the code clone detection tool to comprehensively investigate similar parts and confirmed that there was no problem".

Summary of static analysis tools

You can evaluate the code and find out where to fix it without actually moving the code.

The cost of deploying test code in a conservative environment can be messy, but static analysis tools can be used at low cost and can be used by many organizations.

Story around the test environment

Let me tell you some stories about the test environment. In addition, since the story of integration testing comes out from this chapter, it is becoming a title scam of "system made in C language".

Separation of unit test environment

There is a site where the following configuration is used for unit testing, regardless of whether it is an integration test involving another system. image.png

The bad points of this configuration are as follows. ・ It often happens that the tests that pass each other have an influence and the cases that originally pass are rejected, and vice versa. -Due to the unstable situation of unit testing, it is necessary to change the test environment frequently, and the work of other developers stops each time.

In other words, the testability is extremely poor. Therefore, it is desirable to prepare a test environment for each developer as follows up to the unit test.

image.png

In order to give a test environment to each developer, a long time ago, I used to forcibly build and run a program that runs on Linux with Visual C ++ on Windows. However, these days you can build a virtual environment with VMWare or VirtualBox, so you should use it.

In addition, when the memory of the PC given to the developer is 4GB, I think that I will give up in the virtual environment.

Talking about automating releases to test environments

When releasing to a test environment where multiple people can touch it, it is better to make it as automatic as possible. To minimize the impact of the release, you should reduce the time the test environment is down or release it when no one is using it. This can be dealt with by automating the release process to the test environment.

Also, keeping a record of releases to the test environment allows you to track when and which revision of material was used. You don't have to stick to a specific tool such as Jenkins, you can use a shell script or batch file, so it would be nice to make it automatic.

In addition, by using the technique in the following article, I think that it will be possible to upload the material from Windows with WinSCP, then execute the script via TeraTerm and send the result by e-mail.

** I wonder if it's okay to automate WinSCP ** https://qiita.com/mima_ita/items/35261ec39c3c587210d8

** Try TeraTerm macro ** https://qiita.com/mima_ita/items/60c30a28ea64972008f2

** Automatic operation of Outlook with our PowerShell that gave up Redmine ** https://qiita.com/mima_ita/items/37ab711a571f29346830

Automation of mass data creation during integration testing

You may need to create a large amount of data during the integration test. At this time, from the viewpoint of integration test, if there is a statement that "the data to be used can be created by the system", it is not possible to flow the data into the database with SQL.

To operate the screens provided by the system and register a large amount of data, you can use the technique introduced in "Automating Windows screen operations" below.

** A self-proclaimed IT company went into the field without using IT too much I will introduce a method of automating Windows ** https://qiita.com/mima_ita/items/453bb6c313e459c44689

However, due to the characteristics of the tool to be adopted and the application to be operated, there are cases where input errors etc. can be bypassed, so it is necessary to be careful when adopting it.

Is it possible to automate the screen operation of the integration test?

I think you should narrow down the target for the following reasons.

-Due to the characteristics of the tools used and the application to be operated, there are cases where automatic operation is not exactly the same as manual operation. ・ Screen operation automation is basically unstable -Automatic operation of screen operations requires a high creation cost.

If I do it, I will focus on screen operation tests for smoke tests and a large amount of iterative processing (creating large amounts of data and load tests).

Record of defeat

An ideal but useless story

There are some useless stories out there that are ideal but cannot be introduced as a practical matter. Here, I would like to introduce such a useless story.

Talk about introducing test management tools

image.png

image.png

** About TestLink, a tool for managing the test process ** https://qiita.com/mima_ita/items/ed56fb1da1e340d397b9

I've been proposing things like that in various places for 10 years, but basically, I think it's tough in an environment like SIer.

Organizations that are interested in these tools are already using them, and organizations that aren't interested are ** not really interested in test management itself **, so what you should do if you spend a lot of money on deploying tools that are unlikely to win. I think there are others.

The story of introducing a bug tracking system

By introducing a bug tracking system, you will be able to manage the history of bugs, determine the status of who has bugs and how much, and manage the association with the source code.

Unfortunately, I realized that in some situations it is better to focus on basic education, such as "how to write a bug vote," rather than spending time on that theory.

Accumulation of searchable knowledge by Wiki

When considering searchability and history retention, it is preferable to adopt Wiki rather than accumulating knowledge in Office sentences.

However, many IT engineers in certain environments have only used Excel to write sentences, so don't overdo it.

The next best thing to talk about in vain

In the field where all the above ideal stories are wasted, test specifications, bug tracking forms, and knowledge sentences will be written in Excel or Word and managed in a shared folder.

There are two problems with this. The first is the problem of searchability The second is the issue of configuration management.

Think about how to deal with these issues.

Searchability issues

You can search Office texts using some free software. If anything, you can create it with VBS etc.

In addition, I think that it takes time to search Office, so if possible, I think that it is less stressful to adopt software that uses cache.

Configuration management issues

Files that are randomly placed in a shared folder make it unclear when, who, and what modifications they made. Also, it rarely disappears due to erroneous operation.

As a response, use configuration management tools instead of shared folders to manage your documents.

If it is not possible to introduce the configuration management tool, please consider Plan B by referring to the following article.

** What to do if you reincarnate in a world dominated by shared folder teaching ** https://qiita.com/mima_ita/items/ef3588df4d65cf68f745

at the end

I introduced the findings I gained from trying to improve the testing process in a conservative environment over the years. As I wrote in the following article, especially when trying to improve a conservative environment that has passed many years, it is necessary to accept the current situation and find a realistic landing point.

-[How to propose my thoughts](https://needtec.sakura.ne.jp/wod07672/2020/03/29/%e3%81%bc%e3%81%8f%e3%81 % ae% e3% 81% 8b% e3% 82% 93% e3% 81% 8c% e3% 81% 88% e3% 81% 9f% e3% 81% 95% e3% 81% 84% e3% 81% 8d % e3% 82% 87% e3% 83% bc% e3% 81% ae% e6% 8f% 90% e6% a1% 88% e6% 96% b9% e6% b3% 95 /)

In the first place, the line itself that says "Let's improve the test process"

image.png

It is unavoidable to receive such a reaction, so I think it would be nice if it happened to be accepted.

Recommended Posts