Compiler optimization can result in apparently sound defensive code being eliminated where it is associated with “infeasibility”—that is, where it exists on paths that can’t be tested and verified by any set of possible input values. Even more alarmingly, defensive code shown to be present during unit testing may well be eliminated when the system executable is constructed. Just because coverage of defensive code has been achieved during unit test therefore doesn’t guarantee that it is present in the completed system.

That is why object code verification (OCV) represents best practice for any system for which there are dire consequences associated with failure—and indeed, for any system where only best practice is good enough.

Before and after compilation

Verification and validation practices championed by functional safety, security, and coding standards such as IEC 61508, ISO 26262, and IEC 62304 place considerable emphasis on showing how much of application source code is exercised during requirements based testing.

Experience has shown us that if as much of the code base as possible has been shown to perform correctly then the probability of failure in the field is considerably lower. And yet because the focus is on the high-level source code (no matter what the language), such an approach places a great deal of faith in the ability of the compiler to create object code that reproduces precisely what the developers intended – either because the compiler is flawed, or because of a misunderstanding on the part of the developer. In the most critical of applications, that implied assumption cannot be justified.

Developer intent versus executable behaviour

Compilers are generally highly reliable applications, and while there may be bugs as in any other software, a compiler’s implementation will generally fulfil its design requirements. The problem is that those design requirements do not always reflect the needs of a functionally safe system.

The rules followed by C or C++ compiler vendors permit their products to modify the code in any way they like, provided the binary behaves “as if it were the same” in the context of the assumed environment. But that assumed environment be subtly different to the world of functional safety. Consider Figure 1 which illustrates an example resulting from compilation with the CLANG compiler.

Figure 1

The defensive call to the ‘error’ function has not been expressed in the assembler code.

The compiler has also decided that because the values of the actual objects (13 and 23) are not used in a numeric context, it will use the values of 0 and 1 to toggle between states and then use an exclusive “or” to update the state value. The binary adheres to the “as if” obligation and the code is fast and compact. Within its terms of reference, the compiler has done a good job.

This behaviour has implications for “calibration” tools that use the linker memory map file to access objects indirectly, and for direct memory access via a debugger. Such considerations are not part of the compiler’s assumed environment.

Now suppose the code remains unchanged, but its context in the code presented to the compiler changes slightly (Figure 2).

Figure 2

There is now an additional function “f”, which returns the value of the state variable as an integer. The absolute values 13 and 23 now matter. Even so, those values are not manipulated within the update function (which remains unchanged). The compiler continues to make (valid) value judgements about where the values of 13 and 23 should be used.

If the new function is changed to return a pointer to our state variable, the assembler code changes substantially (Figure 3). Because there is now the potential for alias accesses through a pointer, the compiler can no longer deduce what is happening with the state object. It cannot conclude that the values of 13 and 23 are unimportant and so they are now expressed explicitly within the assembler.

Figure 3

Implications for source code unit test

Now consider the example in the context of an imaginary unit test harness (Figure 4). As a consequence of the need for a harness to access the code under test, the value of the state variable is manipulated and so the default is not “optimized away”.

Figure 4

This is entirely justifiable in a test tool that has no context relating to the remainder of the source code and that is required to make everything accessible, but as a side effect it can disguise the legitimate omission of defensive code by the compiler.

The compiler recognizes that an arbitrary value is written to the state variable via a pointer, and again, it cannot conclude that the values of 13 and 23 are unimportant so they are expressed explicitly within the assembler. On this occasion it cannot conclude that S0 and S1 represent the only possible values for the state variable, so the default path may be feasible. The manipulation of the state variable achieves its aim and the call to the error function is now apparent in the assembler.

However, this manipulation will not be present in the code that will be shipped within a product, and neither will the call to error().

Best practice in any world

If the compiler handles code differently in the test harness compared to the unit test, then is source code unit test coverage worthwhile? The answer is a qualified “yes.” Many systems have been certified on the evidence of such artefacts, and proven safe and reliable in service. But for the most critical systems across all sectors, if the development process is to withstand the most detailed scrutiny and adhere to best practice, then source level unit test coverage must be supplemented by Object Code Verification (OCV). It is reasonable to assume that the compiler fulfils its design criteria, but those criteria do not include functional safety considerations. OCV currently represents the most assured approach where compiler behaviours conform with their specifications, but their specifications are not best suited to the world of functional safety.

Perhaps the most significant issue is the expectation by many developers that a program can be used to detect memory corruption. It would be more reasonable to expect a critical system to do this in hardware (ECC memory, memory parity) and use a trap to handle memory corruption.