Testing High-Integrity Real-Time Software
We want to know whether our software behaves correctly, right? Especially if it’s high-integrity or safety-critical real-time software.
What does behaving correctly mean? Ideally, that’s when the software does what is expected of it. But meeting expectations is pretty tricky, since they may vary depending on the person you discuss them with, so instead we specify requirements and ensure that the software does what is required of it. This process is mandated for high-integrity and safety-critical systems.
It makes sense to write the acceptance criteria in conjunction with, and at the same time as, the requirements. By doing so, we are more likely to write testable requirements.
It also makes sense to write the acceptance test scripts before actually implementing the solution. By doing so, we will know that when the product passes all of its acceptance tests, it is complete. This is Acceptance Test Driven Development (ATDD). I recommend a set of passing acceptance tests and a set of failing acceptance tests. During development, tests are added to the failing set as more features are specified, and tests are moved from the failing set to the passing set as development progress is made. Eventually, the failing test set is empty and the product is complete and does what is required of it.
It is possible to provide tests at the architectural and design levels, but these tests are often covered by the tests mentioned above and below, making them redundant and an expensive luxury.
It is possible and highly desirable to write tests during code development. Ideally, these are the Test Driven Development (TDD) tests: write a failing test; implement a solution until the test passes; commit to the repository.
Some viable system tests may be specified by safety-critical requirements to be included in the implementation, in order to confirm the viability of the hardware or software, for example in aircraft whose computer systems are frequently exposed to cosmic rays.
Testing Frequency
The passing ATDD and TDD tests form the complete regression test suite. At the very least, we should run these tests, confirm that they all pass, and log all the results before each product release.
It is highly desirable to do that before merging each development branch back into the trunk.
The complete regression test suite could also be run following every build, if the host machine is fast enough. If that’s not practical, can a subset of high-speed tests be run?
As embedded processors become ever more powerful, it is helpful to include a subset of the TDD tests into the Power-On-Self-Test (POST). On one system I worked on, towards the end of the project, it would take almost 2 minutes to run the POST before the program actually started its main-loop, but I thought that the trade-off was well worth it. The TDD tests were switched out of the POST later in the project. If the processor had been fast enough, I would have preferred to leave them in.
If the processor is powerful enough compared with the tasks it has to perform, a subset of the TDD tests can be run during idle-time.
Lastly, in safety-critical systems, tests on the viability of the system hardware are frequently run continually, for example checking safety-critical outputs are functioning. This can be extended to the viability of the system software, for example triple redundant variables, confirming memory checksums, message confirmation in inter-processing communication, etc.
In summary: to perform adequate testing of safety-critical real-time software, strive to ensure that requirements are complete and testable; ATDD tests are comprehensive and pass; TDD tests are comprehensive and pass; POST, idle time and run time tests are comprehensive.
You are welcome to contact me to discuss this at plaxion@live.co.uk.
