Testing High-Integrity Real-Time Software

Red = Ruinous, Green = Great

What does behaving correctly mean? Ideally, that’s when the software does what is expected of it. But meeting expectations is pretty tricky, since they may vary depending on the person you discuss them with, so instead we specify requirements and ensure that the software does what is required of it. This process is mandated for high-integrity and safety-critical systems.

It makes sense to write the acceptance criteria in conjunction with, and at the same time as, the requirements. By doing so, we are more likely to write testable requirements.

It also makes sense to write the acceptance test scripts before actually implementing the solution. By doing so, we will know that when the product passes all of its acceptance tests, it is complete. This is Acceptance Test Driven Development (ATDD). I recommend a set of passing acceptance tests and a set of failing acceptance tests. During development, tests are added to the failing set as more features are specified, and tests are moved from the failing set to the passing set as development progress is made. Eventually, the failing test set is empty and the product is complete and does what is required of it.

It is possible to provide tests at the architectural and design levels, but these tests are often covered by the tests mentioned above and below, making them redundant and an expensive luxury.

It is possible and highly desirable to write tests during code development. Ideally, these are the Test Driven Development (TDD) tests: write a failing test; implement a solution until the test passes; commit to the repository.

Some viable system tests may be specified by safety-critical requirements to be included in the implementation, in order to confirm the viability of the hardware or software, for example in aircraft whose computer systems are frequently exposed to cosmic rays.

Testing Frequency

The passing ATDD and TDD tests form the complete regression test suite. At the very least, we should run these tests, confirm that they all pass, and log all the results before each product release.

It is highly desirable to do that before merging each development branch back into the trunk.

The complete regression test suite could also be run following every build, if the host machine is fast enough. If that’s not practical, can a subset of high-speed tests be run?

As embedded processors become ever more powerful, it is helpful to include a subset of the TDD tests into the Power-On-Self-Test (POST). On one system I worked on, towards the end of the project, it would take almost 2 minutes to run the POST before the program actually started its main-loop, but I thought that the trade-off was well worth it. The TDD tests were switched out of the POST later in the project. If the processor had been fast enough, I would have preferred to leave them in.

If the processor is powerful enough compared with the tasks it has to perform, a subset of the TDD tests can be run during idle-time.

Lastly, in safety-critical systems, tests on the viability of the system hardware are frequently run continually, for example checking safety-critical outputs are functioning. This can be extended to the viability of the system software, for example triple redundant variables, confirming memory checksums, message confirmation in inter-processing communication, etc.

The contribution of frequent testing to robustness

In summary: to perform adequate testing of safety-critical real-time software, strive to ensure that requirements are complete and testable; ATDD tests are comprehensive and pass; TDD tests are comprehensive and pass; POST, idle time and run time tests are comprehensive.

You are welcome to contact me to discuss this at plaxion@live.co.uk.

--

--

--

Experienced high-integrity software engineer, crypto code compactor and Datona Labs founder — providing smart contracts to protect your digital information.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Chapter 3 - Source Lines of code: Project Management Course -AUG Unversity

How Business Analysts can enable and participate in Test Driven Development

Image shows the Test Driven Development Cycle, starting with a failing test, making that test pass and then refactoring the code using test to ensure everything still works.

Auto Tweet to Your Discord Server, and Other Interesting Discord + Twitter Integrations

Laravel Advance | How to Add Foreign Key in Laravel Migration

AZURE AD APP REGISTRATION — CREATE APPLICATION USING MS GRAPH API AND POWERSHELL

Write Docker file in Simple way.

AWS RDS with MySQL using Flask

What Php Frameworks You Should Use in 2021?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Jules Goddard

Jules Goddard

Experienced high-integrity software engineer, crypto code compactor and Datona Labs founder — providing smart contracts to protect your digital information.

More from Medium

Database Replication for the developers

Write less but clean quality code.

Why you should be using SOLID

9 Reasons To Go With Java Web Development Services!