Structural coverage analysis, also known as code coverage analysis, provides a measure of the extent to which software code has been exercised under specific test conditions. It generally refers to source code, but it can also reference object code verification – a technique used where applications are especially critical, especially in the aerospace and space sectors.
Structural (or code) coverage is a dynamic analysis metric that describes the extent to which embedded software code has been exercised during test. There are many different metrics for structural coverage – some much easier to achieve than others – but the easier they are to achieve, the less the code under test is exercised in their achievement.
These different metrics are often referenced by standards concerned with functional safety (including DO‑178C, IEC 61508, ISO 26262, IEC 62304, EN 50128…) to ensure that the effort required to achieve adequate structural coverage is proportionate to the criticality of the code under test. The use of automated test tools is not made obligatory by any of these standards, but in most cases, they represent the most economically prudent option for all but very simple applications.
There are several ways of analysing structural coverage. LDRA testing tools analyse the code structure by means of static analysis, and then insert code coverage instrumentation into a copy of the source code to identify which parts of the code have been exercised.
Other methods of analysing structural coverage include the use of real-time trace tools such as those supplied by our technology partners Lauterbach Development Tools .
Structural (or code) coverage purely assesses which parts of the code have been exercised. If the duration of that execution is significant, then WCET (Worst Case Execution Time) is an important metric to consider alongside it.
For many, the answer to that question might be “because the standards say we must”! But the fact is that so many standards wouldn’t insist on it unless it was significantly beneficial.
Pragmatically, structural code coverage is a dynamic analysis technique that provides a measure of how much testing has been completed in a form that can be verified. The implication is that if sufficient testing has been completed, then a proportionate amount of the code can be assumed to be bug-free. Although the targets of the structural coverage do not necessarily have direct correlation to functional bugs, achieving a high structural coverage is likely to increase the chance of revealing problems.
Structural coverage also has an important part to play with regards to bidirectional requirements traceability, as supported by the TBmanager component of the LDRA tool suite.
Of course, it can be used to show that all requirements have been tested. But less obviously, if all requirements are proven to be fully implemented and there is code yet to be exercised, then that code needs to be eliminated or justified.
Static analysis involves examining the code without executing it. It focuses on the control structures, syntax, and other static properties of the code, and provides the understanding of those properties that is required before dynamic analysis can take place. It allows code instrumentation to be correctly placed, and subsequent dynamic analysis to be put into context.
Code coverage instrumentation (also known as software instrumentation or code instrumentation) involves adding extra code to a copy of an application for monitoring program behaviour during dynamic analysis.
There are a variety of mechanisms available for extracting coverage data from the target device under test. These include serial port, TCP/IP port, USB, and JTAG.
Adding code instrumentation to a code base increases its size, which can be a challenge when resources are tight. Just one way of addressing that issue is by using multiple test runs, subdividing the code base such that different sections are instrumented each time.
There are many structural (code) coverage metrics in regular use. Some are favoured more by particular industries, and by the standards associated with them. If standards adherence is not the primary concern, ensuring that the confidence in the code is proportionate to the consequences of its failure is a common sense approach. The following list is roughly in order of rigour, although that is a simplistic perspective. There is more to it than that, as the descriptions clarify.
Function coverage is a measure of the number of functions (sub-programs) that have been called during the execution of a program, as a percentage of the number of functions that exist in that program.
Call coverage is a measure of the number of calls to functions (sub-programs) that have been exercised during the execution of a program, as a percentage of the number of calls that exist in that program.
100% function coverage doesn’t imply 100% call coverage. The opposite is not necessarily true either. Moreover, it is not possible to infer a function coverage percentage from a call coverage metric alone.
ISO 26262:6 §10.4.5 recommends the use of both function coverage and call coverage for less critical applications (ASIL A & B), and highly recommends them where criticality is higher (ASIL C & D).
The illustration below shows an extract from the automotive standard ISO 26262:
Line coverage is a measure of the number of lines that have been exercised during the execution of a program, as a percentage of the number of lines that exist in that program. The achievement of 100% line coverage would imply that all lines in the source have been executed at least once during test.
Statement coverage is a measure of the number of statements that have been exercised during the execution of a program, as a percentage of the number of statements that exist in that program. The achievement of 100% statement coverage would imply that all statements in the source have been executed at least once during test.
Branch coverage (also known as decision coverage) is a measure of the number of branches that have been exercised during the execution of a program, as a percentage of the number of statements that exist in that program. A “branch” is one of the possible execution paths the code can take after a decision statement—e.g., an if statement—gets evaluated.
Combining statement and branch coverage ensures that all statements are exercised fully – including both outcomes of any statement containing a decision.
The extract below is from a table in the aviation standard DO-178C.
It can take a lot of tests to ensure that all possible paths through a multiple condition decision have been exercised. MC/DC coverage is designed to alleviate that, and it does not require every possible combination to be executed. If “n” is the number of conditions, then a minimum of “n + 1” combinations are required to achieve 100% coverage, as opposed to 2n total combinations.
This only really becomes useful when there are for 4 or more conditions, because the number of possible combinations increases exponentially.
The extract below is adapted from a table in the industrial applications standard IEC 61508.
Linear Code Sequence and Jump (LCSAJ) coverage is a measure of the number of LCSAJs that have been exercised during the execution of a program, as a percentage of the number of LCSAJs that exist in that program.
LCSAJ coverage is the most thorough of the attainable source code coverage metrics. It is attainable in that the number of LCSAJs in a code base makes it practical and proportionate to exercise a large majority of them. And it is thorough in that achieving any given level of LCSAJ coverage exercises more of the code base than achieving the same level using a comparable coverage metric (including function, statement, and branch coverage).
An LCSAJ (or a JJ-path) is a path fragment – a linear sequence of executable code. Each LCSAJ commences either from the start of the program or from a point to which control flow may jump. It is terminated either by a specific control flow jump or by the end of the program.
The extract below is adapted from a section of the railways standard EN 50128.
As previously stated, the justification for the use of code coverage analysis in general is that if sufficient testing has been completed, then a proportionate amount of the code can be assumed to be bug-free.
But source code isn’t executed by computers – and what’s more, it doesn’t always resemble the assembly code that is executed. For that reason, the more demanding applications in the aviation and space sectors demand the justification of any discrepancies. Object Code Verification represents the most complete and effective way of achieving that aim.
The extract below is adapted from the space application standard ECSS-E-ST-40C
Many regulatory authorities now require structural coverage analysis to prove that code has been adequately tested, and to measure software test effectiveness. For safety-critical software development, certification bodies need reports that detail the extent of code coverage. And as always, commercial pressures demand that coverage information is collated efficiently, with as little impact on development as possible.
For simple projects, manual- or in-house techniques may be feasible. But to collate the coverage data described above, automating the process is likely to be quicker, more efficient, and more cost effective.
LDRA dynamic analysis tools for embedded systems include:
LDRA testing tools automatically generate test cases, execute those test cases, and visually report levels of coverage analysis, such as statement, branch/decision, function call, LCSAJ (JJ-path), MC/DC, dynamic data flow, and more.
The programming languages supported by LDRA testing tools include C, C++, Java, Ada and Assemblers, running on a broad range of target platforms—from powerful 64-bit microprocessors to highly constrained 8- or 16-bit microcontrollers. The TBrun component of the LDRA tool suite can automatically generate test cases that provide 50-80% of coverage. And its intuitive test case building environment lets developers quickly augment those test cases to increase their coverage if necessary.
Technical white paper: Implementing IEC 61508:2010 with the LDRA tool suite®
Technical white paper: Implementing the EN 5012X standards with the LDRA tool suite®
Technical white paper: Implementing ISO 26262 second edition with the LDRA tool suite®
Technical white paper: Object Code Verification: Why it Matters
Technical white paper: Verification of Airborne Software in Compliance with DO-178C
Email: info@ldra.com
EMEA: +44 (0)151 649 9300
USA: +1 (855) 855 5372
INDIA: +91 80 4080 8707
