2022-07-10 - automatic test groups

we all do automated tests. there are however a lot of differences in how test binaries are organized in the project. some of the patterns i've seen in the past are:

• unit tests, module tests and integration tests are kept in 3 separate binaries.
• there is a “automated tests” binary per module.
• there are separate UTs, MTs and ITs binaries for each module.
• there are multiple separate binaries with tests of each element of module(s).

split into UTs, MTs and ITs is very common. after all this is how we split tests in v-model. making this split also per-module is common to make builds faster (UTs build cycles are short). this is however not the best pattern out there, as it misses the key UX requirement from devs – test execution time! truly – when i work on functionality X, i do not really care if the test is UT, MT or IT, as long as i can execute it fast! slow-running tests i can run later, once i'm sure i've polished out everything on with fast tests. so the natural split is:

• fast (running) tests
• slow (running) tests

to make things elastic and well encapsulated, binaries shall also be created on a per-module basis.

this leaves us with final test group – manual test binaries. these are binaries, that cannot be easily automated. eg. tests that need special HW components connected (like physical port I/O, to talk to 3rd party module), or output processing is not easy to automate, yet easy to verify visually (eg. edge detection on a real-life image). i've talked a bit more about these during my the taming of the Software: C++, embedded... and more tech talk – please watch it, if you're interested in some more details here. for the sake of discussion here we'll just create a 3rd group of tests: manual tests.

to put it a bit more into c++ context, i'm now shifting towards following naming standard:

• fast (running) tests – *.ft.cpp
• slow (running) tests – *.st.cpp
• manual tests – *.mt.cpp

so now for each module i get 1 or more binaries: fast tests, slow tests and some number of manual tests (each binary helps to tests a single feature). for the typical code, we should be done with just “fast tests”. some may include “slow test” (eg. external components setup, or CPU-intensive testing) and some, mostly low-level HW features testing in embedded world (and maybe some visuals), will result in “manual test”.

now running tests is simple and practical. daily, during development, you run “fast tests”. once you think you're done with coding, you run “slow tests” to confirm if you're on track. “manual tests” would be typically touched only when you change the exact functionality, that the test address (eg. RS-232 manual I/O test would only be ran, if implementation of RS-232 stack is changed).