Structural Coverage Analysis (SCA), also known as code coverage analysis, provides a measure of the extent to which software code has been exercised under specific test conditions. It generally refers to source code, but it can also reference object code verification – a technique used where applications are especially critical, especially in the aerospace and space sectors.
In safety-critical software development (and hence in standards such as DO‑178C, IEC 61508, ISO 26262, IEC 62304, EN 50128…) structural coverage analysis is used to confirm the completeness of Requirements-Based Testing (RBT).
A requirements-based system is one where the development process is entirely driven by clearly defined requirements. These requirements describe what the system must do, including both functional behaviours and non-functional properties such as timing, safety, and reliability. Throughout the system’s life cycle, every design decision, implementation step, and test case is linked directly to a specific requirement.
Testing in a requirements-based system is focused on demonstrating that each requirement has been properly implemented and verified. It also serves to confirm that the software contains no additional, unintended functionality beyond what is necessary to meet the requirements.
In short – no. As Agile and other iterative methods become more common in safety-critical development, requirements-based testing is adapted to fit these workflows. User stories are treated as formal requirements, and verification activities are carried out incrementally as new code is developed and tested. In this way, testing ensures that all implemented functionality is properly exercised and traceable to requirements, just as it would be in a traditional V-model.
At release or certification milestones, cumulative evidence must still demonstrate complete and justified verification, maintaining the same level of rigor expected in more conventional development processes.
Requirements-based testing typically progresses through five distinct stages:
Testing concludes only when all functional and non-functional requirements have been fully verified.
Each test case is defined by five key elements: the initial state or precondition, data setup, inputs, expected outcomes, and actual outcomes.
Test cases are executed against the system under test, with all results carefully documented.
A test is deemed successful when its actual results align precisely with the expected outcomes.
It must be confirmed that testing adequately covers both the functional and non-functional aspects of all requirements.
Following this process will ensure that code has been implemented to address every system requirement, that the implemented code has been tested to completeness, and there is no code that is not attributable to a requirement. It is possible to achieve bidirectional traceability using traditional methods, but automating the process with the TBmanager component of the LDRA tool suite (below) makes it is much less error prone and labour intensive.
Structural coverage analysis and requirements-based testing are closely related, but not always inseparable. In safety-critical domains, structural coverage is used specifically to confirm that requirements-based tests adequately exercise the code. In general software development, however, structural coverage may be applied independently of requirements, serving more broadly as a quality assurance measure.
For many, the answer to that question might be “because the standards say we must”! But the fact is that so many standards wouldn’t insist on it unless it was significantly beneficial.
Pragmatically, structural code coverage is a dynamic analysis technique that provides a measure of how much testing has been completed in a form that can be verified. The implication is that if sufficient testing has been completed, then a proportionate amount of the code can be assumed to be bug-free. Although the targets of the structural coverage do not necessarily have direct correlation to functional bugs, achieving a high structural coverage is likely to increase the chance of revealing problems.
Structural coverage also has an important part to play with regards to bidirectional requirements traceability, as supported by the TBmanager component of the LDRA tool suite.
Of course, it can be used to show that all requirements have been tested. But less obviously, if all requirements are proven to be fully implemented and there is code yet to be exercised, then that code needs to be eliminated or justified.
Structural (or code) coverage purely assesses which parts of the code have been exercised. If the duration of that execution is significant, then WCET (Worst Case Execution Time) is an important metric to consider alongside it.
Static analysis involves examining the code without executing it. It focuses on the control structures, syntax, and other static properties of the code, and provides the understanding of those properties that is required before dynamic analysis can take place. It allows code instrumentation to be correctly placed, and subsequent dynamic analysis to be put into context.
Code coverage instrumentation (also known as software instrumentation or code instrumentation) involves adding extra code to a copy of an application for monitoring program behaviour during dynamic analysis.
There are a variety of mechanisms available for extracting coverage data from the target device under test. These include serial port, TCP/IP port, USB, and JTAG.
Adding code instrumentation to a code base increases its size, which can be a challenge when resources are tight. Just one way of addressing that issue is by using multiple test runs, subdividing the code base such that different sections are instrumented each time.
There are many structural (code) coverage metrics in regular use. Some are favoured more by particular industries, and by the standards associated with them. If standards adherence is not the primary concern, ensuring that the confidence in the code is proportionate to the consequences of its failure is a common sense approach. The following list is roughly in order of rigour, although that is a simplistic perspective. There is more to it than that, as the descriptions clarify.
Function coverage is a measure of the number of functions (sub-programs) that have been called during the execution of a program, as a percentage of the number of functions that exist in that program.
Call coverage is a measure of the number of calls to functions (sub-programs) that have been exercised during the execution of a program, as a percentage of the number of calls that exist in that program.
100% function coverage doesn’t imply 100% call coverage. The opposite is not necessarily true either. Moreover, it is not possible to infer a function coverage percentage from a call coverage metric alone.
ISO 26262:6 §10.4.5 recommends the use of both function coverage and call coverage for less critical applications (ASIL A & B), and highly recommends them where criticality is higher (ASIL C & D).
The illustration below shows an extract from “Table 12 – Structural coverage at the software architectural level” in the automotive standard ISO 26262:
Line coverage is a measure of the number of lines that have been exercised during the execution of a program, as a percentage of the number of lines that exist in that program. The achievement of 100% line coverage would imply that all lines in the source have been executed at least once during test.
Statement coverage is a measure of the number of statements that have been exercised during the execution of a program, as a percentage of the number of statements that exist in that program. The achievement of 100% statement coverage would imply that all statements in the source have been executed at least once during test.
Branch coverage (also known as decision coverage) is a measure of the number of branches that have been exercised during the execution of a program, as a percentage of the number of statements that exist in that program. A “branch” is one of the possible execution paths the code can take after a decision statement—e.g., an if statement—gets evaluated.
Combining statement and branch coverage ensures that all statements are exercised fully – including both outcomes of any statement containing a decision.
The extract below is from Table A-7 “Verification of Verification Process Results” in the aviation standard DO-178C.
It can take a lot of tests to ensure that all possible paths through a multiple condition decision have been exercised. Multiple Condition/Decision Coverage (MC/DC) is designed to alleviate that, and it does not require every possible combination to be executed. If “n” is the number of conditions, then a minimum of “n + 1” combinations are required to achieve 100% coverage, as opposed to 2n total combinations.
This only really becomes useful when there are for 4 or more conditions, because the number of possible combinations increases exponentially.
The table below is adapted from Table B.2 “Dynamic analysis and testing” in the industrial applications standard IEC 61508.
Linear Code Sequence and Jump (LCSAJ) coverage is a measure of the number of LCSAJs that have been exercised during the execution of a program, as a percentage of the number of LCSAJs that exist in that program.
LCSAJ coverage is the most thorough of the attainable source code coverage metrics. It is attainable in that the number of LCSAJs in a code base makes it practical and proportionate to exercise a large majority of them. And it is thorough in that achieving any given level of LCSAJ coverage exercises more of the code base than achieving the same level using a comparable coverage metric (including function, statement, and branch coverage).
An LCSAJ (or a JJ-path) is a path fragment – a linear sequence of executable code. Each LCSAJ commences either from the start of the program or from a point to which control flow may jump. It is terminated either by a specific control flow jump or by the end of the program.
The railways standard EN 50716 states “Based on an analysis of the program a set of input data is chosen such that a large fraction of selected program elements are exercised. The program elements exercised can vary depending on the level of rigor required”
The standard lists elements for consideration including LCSAJ, of which it says: “A linear code sequence and jump is any linear sequence of code statements including conditional jumps terminated by a jump.”
As previously stated, the justification for the use of code coverage analysis in general is that if sufficient testing has been completed, then a proportionate amount of the code can be assumed to be bug-free.
But source code isn’t executed by computers – and what’s more, it doesn’t always resemble the assembly code that is executed. For that reason, the more demanding applications in the aviation and space sectors demand the justification of any discrepancies. Object Code Verification represents the most complete and effective way of achieving that aim.
The extract below is adapted from the space application standard ECSS‑E‑ST‑40C §5.8.3.5e which states “In case the traceability between source code and object code cannot be verified (e.g. use of compiler optimization), the supplier shall perform additional code coverage analysis on object level code as follows”
Many regulatory authorities now require structural coverage analysis to prove that code has been adequately tested, and to measure software test effectiveness. For safety-critical software development, certification bodies need reports that detail the extent of code coverage. And as always, commercial pressures demand that coverage information is collated efficiently, with as little impact on development as possible.
For simple projects, manual- or in-house techniques may be feasible. But to collate the coverage data described above, automating the process is likely to be quicker, more efficient, and more cost effective.
LDRA dynamic analysis tools for embedded systems include:
LDRA testing tools automatically generate test cases, execute those test cases, and visually report levels of coverage analysis, such as statement, branch/decision, function call, LCSAJ (JJ-path), MC/DC, dynamic data flow, and more.
The programming languages supported by LDRA testing tools include C, C++, Java, Ada and Assemblers, running on a broad range of target platforms—from powerful 64-bit microprocessors to highly constrained 8- or 16-bit microcontrollers. The TBrun component of the LDRA tool suite can automatically generate test cases that provide 50-80% of coverage. And its intuitive test case building environment lets developers quickly augment those test cases to increase their coverage if necessary.
Technical white paper: Implementing IEC 61508:2010 with the LDRA tool suite®
Technical white paper: Implementing the EN 5012X standards with the LDRA tool suite®
Technical white paper: Implementing ISO 26262 second edition with the LDRA tool suite®
Technical white paper: Object Code Verification: Why it Matters
Technical white paper: Verification of Airborne Software in Compliance with DO-178C
Email: info@ldra.com
EMEA: +44 (0)151 649 9300
USA: +1 (855) 855 5372
INDIA: +91 80 4080 8707