Successful Unit Test

Saturday, October 22, 2016

cmocka and cmake

I am using cmocka as the unit test framework to illustrate the concepts used in my book “a Guide to Success Unit Tests”.

cmocka relies on cmake build system to help it build the cmocka dynamic library as well as the test scripts into executables. So cmake will have to be installed as well as cmocka for it work correctly.

cmake can be downloaded from http://www.cmake.org/download/
cmocka can be downloaded from https://cmocka.org/files/

The I am using a MacOS computer, so my references are from using that computer.

Installing CMake

When installing cmake, many different packaging versions can be downloaded. Since my computer is a MacOS, I downloaded the .dmg package. After I have extracted and installed it, I added it path the cmake executable.

export PATH=/Applications/CMake.app/Contents/bin:$PATH

I test that it can find it by getting it to report is version number.

localhost:~ tehnyitchin$ ccmake --version

ccmake version 3.3.0

CMake suite maintained and supported by Kitware (kitware.com/cmake).

localhost:~ tehnyitchin$

Checking cmocka is working

Once you have downloaded cmocka and unpacked it, you can test if the cmake and cmocka are going to work correctly. The file README has the correct instructions. In the example below, I have unzipped cmocka into a directory called cmocka.



cd cmocka

mkdir build

cd build

cmake ../.

make

After the build, the test binaries are in subdirectory example. Executing one of the test binaries will execute the test scripts. Below is a execution of simple_test



localhost:example tehnyitchin$ ./simple_test

[==========] Running 1 test(s).

[ RUN ] null_test_success

[ OK ] null_test_success

[==========] 1 test(s) run.

[ PASSED ] 1 test(s).

localhost:example tehnyitchin$

I am writing a Guide to Successful Unit Tests, you can get it here at Leanpub

Installing gcc on an oldish OSX so I can get code coverage to work

On one my computers that I am using, it is running OSX 10.8.5, this is known as Mountain Lion. On this computer, I want to install the gcc on it, but I have some difficulties in find the right instructions.

The default gcc that is install is an alias to

localhost:bin tehnyitchin$ ll gcc*

lrwxr-xr-x  1 root  wheel  12 Aug 18  2013 gcc -> llvm-gcc-4.2

So it is not really running the real gcc that you can get from GNU. Scouring the internet, I came across these instructions at Helsinki University Geodynamic Group.

https://wiki.helsinki.fi/display/HUGG/Install+for+older+versions+of+Mac+OS+X

I am working on this as I want to get code coverage working. It is not working at all with the default llvm-gcc-4.2. Compiling the code with a --coverage causes nothing to occur.

Typically, when the code is compiled with --coverage, the .gcno file is also generated. gcov uses this file as part of its code coverage measurements. If I enable the -v option to increase its verbosity, I can see that it is linking the gcov library, but I am not sure if it is actually doing it.

This question on Stack Overflow shows that the code coverage on my OSX is not working.
http://stackoverflow.com/questions/8622293/no-code-coverage-with-mac-os-x-lion-and-xcode-4-llvm-g-4-2

As I execute a simple main.c, I am not getting the correct .gcno. However, when I switch it from gcc to clang, the .gcno files are generated.

Back to the story, After downloading xcode-4 as per the instruction from Helsinki, it failed to installed due to some unknown reason. The logs did not show anything obvious. I did not continue with installation as clang was generating the code coverage files for me. My feeling is that Apple has built llvm-gcc-4.2 without code coverage support. To overcome this, the real gcc is needed. Luckily, in my case, clang came to the rescue.

I am writing a Guide to Successful Unit Tests, you can get it here at Leanpub.

Unit Test code and technical debt

Technical debt has many different meaning. It depends on the context that it is used in. Wikipedia has a nice statement that I quite like.

“The debt can be thought of as work that needs to be done before a particular job can be considered complete or proper. If the debt is not repaid, then it will keep on accumulating interest, making it hard to implement changes later on.” – [0]

Unit tests is one area where technical debt can be easily obtained, and the amount of debt can grow exceptionally quickly. It definitely needs to be forethought and well planned before it is used in full force. The following questions must be answered to fully.

What language will the test code be implemented in? – This is critical contributor to technical debt. If you use a language which the majority of your development team is not familiar with, the chances of increasing technical debt is quite high. As the language is not well known by your team, training will need to be provided to it and the future team that will be maintaining the code. For example, implementing the test code in Java while the code under test is in C means that your development team must know Java and C to deliver good software.

Is the infrastructure of your unit test environment well supported? – Using a unit test framework that does not have a well supported community (either through the online community or supported commercially) means that the support of it must be carried out by your development team. When the code enters maintenance phase, the unit test framework still needs to be maintained, as specially as the computers and operations system it is using starts to revised to newer versions. This particularly true if the code has a long life time, as it is true with most embedded software. As more test cases are being implemented, the amount of technical debt gets larger and it makes supporting the unit test framework quite difficult.

Is the unit test framework integrated well with your tool chain? – Using a unit test framework that is not well integrated in your tool chain means that the result from you test runs must be analysed manually. For a small project, manual analysis of the the test result is OK and relatively easy. However as the project gets bigger with a large code set with high level complexity, good integration into your tool chain is a must. This will ensure that the results are easily analysed, and the correct notifications are generated. It will ensure continuous integration can be achieved easily and automatically. It will also means that removing manual work will also reduce the risk of performing the manual process incorrectly.

By answering these questions, there is a chance that your technical debt could become technical credit.

I am writing a Guide to Successful Unit Tests, you can get it here at Leanpub.

References
[0] – https://en.wikipedia.org/wiki/Technical_debt

Should it be the same or different?

The language of the unit test framework is extremely important. It is just as important as the language of the software module. In fact, the test scripts that are written is a piece of code that needs to be managed. So ideally, the principles of software engineering also applies to the test scripts as well.

To reduce the complexity in your test scripts, it would be ideal if the test scripts are written in the same language as the software module under test. If you are writing your software module in C, your test scripts should also be written in C. If you are using rust, then rust should be used for the test scripts.

The advantage of using the same language for both the software module and test script, it is compiled for testing on the target. The big assumption is that the language is already supported by your target. In some cases, this is not. If you are developing using embedded Linux and your test scripts are in written in python, the chance of your version of python being supported on your target is pretty good. However, if you are targeting an 8bit Atmel AVR, getting python support on the target is pretty slim. If the test script is in the same language as the software module for your target, you are pretty certain that test on target is possible.

The other advantage of using the same language is the ability to debug. In a mixed language environment, the chance of debugging your test script as well as your software module is pretty small. For example, the software module is written in C and the test script is also written in C. When the binary is created and executed, it can be easily debugged using gdb or another debugger. It can be debugged from entry into main() until it exits main(). The variables can also be inspected and modified during debugging quite easily. In a mixed language environment, the variables public names may be mangled and that would make it difficult to find for inspection or even to modified.

To sum up, if you have no compelling reason to not keep all of your test script and software module in the single language domain such as C, do not mix your language.

I am writing a Guide to Successful Unit Tests, you can get it here at Leanpub.

Write the code first, test code that is.

In a software development process, certain steps must be taken to ensure that your software module is delivered as bug free as possible. The aim is for it to be working without any bugs. However, the question is what bug free is exactly. One possible definition is the software module is consider bug free if it fulfills all its requirements, functional and non-functional. Beyond that definition, it is hard to control the injection of bugs from outer sources. For example, if the compiler is not generating the code and causes a mis-behaviour of the software module. Technically, this is a bug for the product and will still to be fixed, but it is hard to control if the behaviour of the compiler is not well known.
So with that definition, the steps in the process are clear. First, the requirements are finalised, and then the implementation can start. Since the requirements are finalised, it would make sense for the test code to be written first. From the requirements, the functional behaviour of the software module is well defined. This means that the input data into the software module is known as well as the results from it. If these are not clear or have some doubts, it is a sign that the requirements are not clear enough. The requirements need more work.

Once the test code are written and executed. Obviously, all the test cases will fail as the code for the software module is not written yet. Now is the write time to start writing your code for the software module. As you continue to write the software module, the test cases are used as a validation for the software module. The implementation continues until all your test cases passes.

Suppose the code for software module is written before the test code. The downside is that the chance of getting your software module correct is low and you won’t know it until you have some test code to do the testing. You have just invested into some technical debt. If your test code is written first, it is very likely to be correct the first time as all the inputs, all the outputs and the behaviours are well defined.

Save time and effort by writing the test code first, and then write the code for your software module.

I am writing a Guide to Successful Unit Tests.
You can get it here at Leanpub. or here at Gumroad. and read about these topics and more.

Where are your input values coming from?

When design your test cases, you will have to decide what type of input values are want to use, but consider where your input values are coming form. The most obvious way of inject values are by passing into via the function parameters. It is logical and it is the formal method. However, that is only one of the ways of injecting values into it. The three most popular methods are.

Function Parameters -Function parameters are part of full definitions of the function. It is one of the fundamental ways of passing data into the function. The range of values that it can take are defined by its type. Function Parameters can also be used to get values out of the function. In C, the function parameters live on the stack memory, and its life time ends when the function ends.

Returned Values – The low level functions called by the your module under tests must be considered as an input. From a unit test perspective, the return values can be influenced via the mocked function. The range of values is dependent on its data type. In C, the function parameters live on the stack memory, and its life time ends when the function ends. The other thing to consider is that this method does not necessary inject the values at the start of your function under test.

Global Variables – variables used in the software modules that are accessible outside of the modules. This method is best avoided in complicated code bases as it tends to create a software module that is prone to unpredictable behaviour if the global variables are not controlled well. The global variable lives on the heap, and it is alive as long as the program is alive.

For most parts, the input methods comes down the how you have design your module. Some design guideline forbid the use of mechanism such as global variables, so make sure that your methods matches your design guidelines.

I am writing a Guide to Successful Unit Tests.
You can get it here at Leanpub. or here at Gumroad. and read about these topics and more.

Rust approach to tests

About nine months ago, I started to hear good things about a new language called Rust. It really gain traction for me when I read a blog post by Gregely Imreh (@imrehg) who wrote about it as his language of the month. The two things that I read about Rust are that it is fast and it is safe. Both designed from the outset with these two items in mind. For a system level language, this is great!

I started reading the rust docs and was pleasantly surprise that unit test is a built in design feature of language. I was really surprise at this. The thing with most languages, unit test is an add on feature that is implemented by another vendor, or another group. Look at C, apart from the fact that its behavior is sometimes unpredictable, a unit test framework is not even defined. You will have to use cmocka, or Unity or one of the other many C test frameworks. Same with python, php, Java etc.

All the details about testing on Rust are show in this chapter in this book.
The more I read about Rust, it strikes me the designers are seasoned C language developers, suffered through many projects with the deficiency of C, and setup to design a language with those deficiency out of the language. Rust is looking really good as a replacement for C or C++.

I am writing a Guide to Successful Unit Tests.
You can get it here at Leanpub. or here at Gumroad. and read about these topics and more.