Developing software in accordance with MIL-STD-882E and the Joint Software Systems Safety Engineering Handbook (JSSSEH)
Nearly all defense systems – across the branches of the Department of Defense – have requirements for system safety. DoD Instruction (DoDI) 5000.02, Operation of the Adaptive Acquisition Framework, establishes a risk management program to ensure program cost, schedule, and performance objectives are achieved, and to communicate the process for managing program uncertainty. MIL-STD-882E together with the Joint Software Systems Engineering Handbook (JSSSEH) provide guidance on the design, development, and verification of safe systems and software.
MIL-STD-882E is a system safety standard practice that identifies the Department of Defense (DoD) Systems Engineering (SE) approach to eliminating hazards, where possible, and minimizing risks where those hazards cannot be eliminated. The Standard covers hazards as they apply to systems / products / equipment / infrastructure (including both hardware and software) throughout design, development, test, production, use, and disposal. The standard specifies system safety requirements throughout the life-cycle for any system. When properly applied, these requirements should enable the identification and management of hazards and their associated risks during system developmental and sustaining engineering activities.
Section 4.4 of the standard states –
“The assessment of risk for software, and consequently software-controlled or software-intensive systems, cannot rely solely on the risk severity and probability. Determining the probability of failure of a single software function is difficult at best and cannot be based on historical data. Software is generally application-specific and reliability parameters associated with it cannot be estimated in the same manner as hardware. Therefore, another approach shall be used for the assessment of software’s contributions to system risk that considers the potential risk severity and the degree of control that software exercises over the hardware.”
The approach outlined in the standard starts by establishing 5 Software Control Categories (SCCs) and 4 Severity Categories. These categories are then used to create a Software Safety Criticality Matrix (SSCM) –
The SSCM specifies a Software Criticality Index (SwCI), and then this SwCI is used to specify the Level of Rigor (LOR) tasks that are the minimum set of tasks required to assess the software contributions to the system-level risk.
Appendix B of MIL-STD-882E provides high level guidance on the LOR tasks. Rather than going into details of the tasks, the standard directs the reader to consult the Joint Software Systems Safety Engineering Handbook and the North Atlantic Treaty Organization (NATO) Allied Ordnance Publication (AOP) 52, Guidance on Software Safety Design and Assessment of Munitions Related Computing Systems for additional guidance on how to conduct required software analyses.
The purpose of the Joint Software Systems Safety Engineering Handbook (JSSSEH) is to provide management and engineering guidelines to achieve a reasonable level of assurance that software will execute within the system context with an acceptable level of safety risk. The handbook is a product of a joint effort: the U.S. Army, Department of the Navy, Air Force, and Coast Guard Safety Centers, with cooperation from the Federal Aviation Administration (FAA), National Aeronautics and Space Administration (NASA), defense industry contractors, and academia, are the primary contributors. The handbook captures the “best practices” pertaining to Software System Safety (SSS) program management and safety-critical software design. The handbook is an instructional guide for understanding SSS and the contribution of each functional discipline to the overall goal. The handbook is applicable to all types of systems (military and commercial) in all types of operational uses.
As described above, MIL-STD-882E specifies 5 software criticality levels (SwCI 1 – SwCI 5) where SwCI 1 is the most critical. The JSSSEH also specifies 5 software criticality levels using the same criteria as MIL-STD-882E, however a few terms are slightly different – the SSCM is referred to as a Software Criticality Matric (SCM), and the SwCI is referred to as a Software Criticality Index (SCI). As in MIL-STD-882E, the handbook instructs the user to create a LOR table that specifies the Level of Rigor tasks to be completed throughout the software development lifecycle – the higher the criticality, the more rigorous the tasks.
It is worth noting that unlike commercial functional safety standards such as DO-178C, ISO 26262, and IEC 61508, the JSSSEH does not require specific objectives to be met based on the SCI; instead, it provide a list of example tasks that can be discussed and considered in the development of a LOR table for a specific program. The individual program tailoring of the LOR table is based on the acceptance or certification criteria of the customer. The oversight of the application of the LOR tables must be included in the systems engineering technical reviews and the milestone reviews. Of particular interest are the amount of re-assessment, re-analysis, and re-testing performed for a given change. The amount and scope of regression testing should be driven by safety criticality (and the risk of not performing), and not by budget or schedule.
MIL-STD-882E and the JSSSEH refer to several government documents, handbooks, and standards, including:
DoD Instruction (DoDI) 5000.02, Operation of the Adaptive Acquisition Framework, establishes policy and prescribes procedures for managing acquisition programs, pursuant to the relevant sections of Title 10, United States Code. The framework also establishes a risk management program to ensure program cost, schedule, and performance objectives are achieved, and to communicate the process for managing program uncertainty.
The North Atlantic Treaty Organization (NATO) Allied Ordnance Publication (AOP) 52, Guidance on Software Safety Design and Assessment of Munitions Related Computing Systems is a “how-to” guide for use in the understanding of SSS and the contribution of each functional discipline to the overall goal. It is applicable to all types of weapons and related systems in all types of operational uses. While the focus of this page is on MIL-STD-882E and the JSSSEH, the guidance in AOP-52 is very similar the guidance provided in the JSSSEH.
MIL-STD-498, titled Software Development and Documentation, includes all activities pertaining to software development suitable for the development of both weapon systems and Automated Information Systems. This standard standard merges DOD-STD-2167A and DOD-STD-7935A.
ISO/IEC/IEEE Standard 12207, titled Systems and software engineering — Software life cycle processes, has replaced MIL-STD-498. The standard provides a common process framework for describing the life cycle of systems created by humans, adopting a Software Engineering approach. It considers both the business and the technical needs of all stakeholders with the goal of providing a quality product that meets the needs of users and other applicable stakeholders. It provides the processes for acquiring and supplying systems. In addition, this framework provides for the assessment and improvement of the life cycle processes.
RTCA DO-178C titled “Software Considerations in Airborne Systems and Equipment Certification” provides recommendations for the production of software for airborne systems and equipment. DO-178C focuses on assuring software performs its intended function with a level of confidence in safety. Both DO-178C and MIL-STD-882E/JSSSEH follow safety integrity level (SIL) based software assurance approaches. These approaches assess the severity of the system’s safety-significant functions that are controlled by software. In doing so, each of the software safety-significant functions can be labeled with a software control category for the purpose of defining the level of rigor that will be required in the function’s design, implementation, test, and verification. By and large, the MIL-STD-882E/JSSSEH software safety engineering tasks align with DO-178C process activities, and MIL-STD-882E/JSSSEH level of rigor tasks align with DO-178C objectives.
While MIL-STD-882E focuses on system safety, it refers to the JSSSEH to provide guidance on software system safety engineering. Section 4 of the JSSSEH focuses on the managerial process and the technical methods and techniques inherent in the performance of software system safety tasks within a systems safety engineering and software development program. The LDRA tool suite capabilities support many of the technical methods and tasks described in section 4, of particular interest are methods and tasks in the following sections:
| JSSSEH Task/Technique | How LDRA Can Help |
|---|---|
| 4.3.6.3.3 Traceability Analysis | Bidirectional traceability across requirements, code, and tests. Automated gap analysis and impact analysis. |
| 4.3.7.2.1 Safety Interlocks | Static analysis, compliance with safe and secure coding practices, code quality analysis, data and control flow analysis, taint analysis, mc/dc test case planning. |
| 4.3.7.3.2 Code Level Analysis | Static analysis, compliance with safe and secure coding practices, code quality analysis, exclusion management, data and control flow analysis, taint analysis. Object code verification for cases where compiler optimizations can change the control structure of the source code. |
| 4.4.1.2 Nominal and Functional Requirements-Based Testing 4.4.1.3 Robustness Testing 4.4.1.4 Requirements Coverage Analysis 4.4.1.5 Structural Coverage Analysis 4.4.1.6 Formal Safety Qualification Testing | Unified test management across all testing techniques. Bidirectional traceability across requirements, code, tests, activities/tasks/objectives. Centralized report management and project-wide test results and analysis report aggregation, along with automated trend visualization. TÜV certification of LDRA tool suite verifying it is qualified to be used in safety-related software development. Tool Qualification Support Packages (TQSPs) greatly reduce the time and effort of qualifying the LDRA tool suite for use as a verification tool in a project environment. |
| 4.4.2.1 Requirements-Based Tests 4.4.2.2 Functionality Tests | Automated requirements-based testing to confirm the software functions as required. Testing can be performed on individual requirements and groups of requirements. Interactive visualization and reporting showing which requirements have been verified. |
| 4.4.2.3 Path Coverage Testing 4.4.2.4 Statement Coverage Testing | Complete coverage analysis including statement coverage, branch/decision coverage, Multiple Condition/Decision Coverage (MC/DC), function and call coverage, data and control coupling coverage, Linear Code Sequence and Jump (LCSAJ) Coverage, and Object Code Verification. Justification management for necessary gaps in coverage . |
| 4.4.2.5 Stress Testing | Software execution time analysis, including worst case execution time analysis on single and multi-core processor target hardware. |
| 4.4.2.8 Fault Insertion and Failure Mode Tests 4.4.2.9 Boundary Condition Tests 4.4.2.10 GO/NO-GO Path Tests 4.4.2.11 Mutation Testing 4.4.2.12 Perturbation Testing | Automatic test case generation is used to create robustness tests including fuzz testing, equivalence class testing, and boundary value testing. Test cases are efficiently created to ensure maximum coverage and minimize redundancy. |
| 4.4.2.13 Test Phases | Automated unit, integration, and system level testing of software using virtual and physical target hardware Automatic generation of test harnesses. |
| 4.4.2.14 Regression Testing | Automated regression testing of functions, files, components, and full software systems using virtual and physical target hardware. |
Preliminary software design includes LOR tasks focused on determining how the software development team interpret and implement the software safety requirements (SSRs) in the design architecture. This is an essential element of the total software safety effort.
Traceability analysis is detailed in section 4.3.6.3.3. Tracing encompasses a requirement-to-code trace and a code-to-requirement trace. Traceability Analysis should include the following:
Bidirectional Traceability helps to ensure that all outline requirements (including functional safety requirements) have been completely addressed, that all detailed requirements can be traced to outline requirements, and in the case of software requirements that there are no surplus, spurious code.
The TBmanager component of the LDRA tool suite automatically maintains the connections between the software requirements, development, and testing artefacts and activities, ensuring that any changes in the associated documents or software code are automatically highlighted. Any consequential re-testing can then be dealt with accordingly.
Following preliminary software design, detailed software design includes methods and tasks for the system safety engineer, software developer, and IV&V engineers to ensure that the SSRs have been implemented as intended and that this implementation has not introduced additional safety concerns.
Safety interlocks are described in section 4.3.7.2.1. These include checks on typing, strong typing, isolation and partitioning of safety-critical source code, and ensuring that all possible combinations and code paths for conditional/branching statements have been accounted for, and to avoid any extraneous or undesired code execution. Many of these safety interlocks can be examined using static analysis to verify conformance with coding standards.
Code-level analysis is described in section 4.3.7.3.2. Code-level analysis begins with an analysis of the architecture to determine the flow of the program, calls made by the executive routine, the structure of the modules, the logic flow of each module, and the implementation in the code. Regardless of the technique used to analyze the code, the analyst must first understand the structure of the software, how it interacts with the system, and how it interacts with other software modules. The key for the analyst is to focus on where safety-significant data is converted, modified, created, and used to make safety-significant decisions.
Section 4.3.7.3.2.7 provides high level guidance on the use of code analysis software tools – in place of or in addition to manual inspection – that are often used to assist in the evaluation of the correctness of the code. This section includes a word of caution about compiler optimizations that can change the control structure of the source code, thus making object code verification very challenging when required as an LOR task.
The static analysis capabilities of the LDRA tool suite provide a means of analysis to address many aspects of safety interlocks and code level analysis. Static analysis can be likened to an automated “inspection” of the source code, comparing the code under review with the chosen software coding standard and deriving code quality metrics. Non-conformances are highlighted as required, along with other undesirable characteristics such as high complexity.
TBexclude provides a multi-tier coding violation exclusion capability enabling the suppression of rule violation reports at the project, team, or individual user levels.
Data-flow analysis is a technique for gathering information about the possible set of values calculated at various points in a computer program. It uses information derived from control-flow analysis to determine how values assigned to variables might propagate. Control-flow analysis focuses on those decision points to provide a more intuitive representation of that behavior, known as control-flow graphs (or control-flow diagrams). In turn, that helps developers to ensure that the code they have written fulfils their intentions.
TBvision provides the core data-flow analysis and control-flow analysis capabilities for embedded software, and a visualization engine to easily understand and navigate the results of those analyses.
LDRA verification tools also provide data and control coupling metrics. The intent is to show that the software modules affect one another only in the ways intended by the software design, ensuring that there are no unplanned, anomalous, or erroneous behaviors.
Object Code Verification (OCV) provides a comprehensive approach to the demonstration of source code to object code traceability. The TBobjectbox component of the LDRA tool suite provides a complete Object Code Verification (OCV) solution, including source code to object code traceability and object code coverage analysis.
At the beginning of a program, it is important to reach an agreement between the certifying safety organization, the program management office, and the system developer as to the depth of software V&V activities that must be performed to provide the desired level of confidence that the software will function as expected. The SSS team should integrate safety testing into the normal system testing effort to reduce time and cost to the program.
In particular, regression testing shall be defined for the program. The extent of regression coverage of existing SSRs within any new baseline, increment, or since last regression shall be incorporated into milestone decisions. The software safety engineers, in cooperation with the V&V team, will identify the SSRs that can be verified through testing.
Section 4.4.1.1 provides overall guidelines for meeting the testing objectives of safety-critical software testing. Sections 4.4.1.2 – 4.4.1.5 characterize different types of testing that may be included in the test plans to satisfy LOR tasks. Types of testing include functional testing, robustness testing, coverage analysis, and formal safety qualification testing.
Section 4.4.1.6 focuses on planning for formal software qualification when a specific suite of test cases are used to provide evidence to the SSS team for safety certification. The agreed testing plan can be described in terms of a set of testing objectives and activities. For the formal software qualification, evidence and artifacts proving the successful completion of the objectives and activities is essential and must be included in the Safety Assessment Report (SAR).
Section 4.4.1.6 also provides high-level guidance on tool qualification – “The software test planning process…must address all simulators, models, emulators, and software tools that will be used by the test team (whether for V&V or safety testing) to ensure that all processes, requirements, and procedures for validation are in place. This validation must occur prior to use to ensure that the data is processed as intended. Invalid tools will invalidate the test results.”
The TBmanager component of the LDRA tool suite is a desktop traceability application, and is integrated with code review, data and control coupling analysis, low-level testing, and code coverage tools. It supports bidirectional traceability not only of project requirements, but also of functional safety and standards objectives and activities, along with associated reports and evidence that capture the result from performing the objectives and activities.
LDRAvault is complementary to TBmanager. It is a web-based, enterprise level application that aggregates and manages certification artifacts across projects and programs providing transparency into the development and verification process. Data access can be easily controlled, and analysis and compliance results can be easily shared with pertinent parties including regulatory authorities. Examples include code reviews, code coverage analysis, and unit testing results.
SGS TÜV SAAR and TÜV SÜD are both independent certifying agencies who have evaluated LDRA’s development and testing practices and issued certificates for the tool suite verifying it is qualified to be used in safety-related software development. Their validation of LDRA tools is adequate assurance of their suitability for most projects.
For more demanding projects that require project specific tool qualification, Tool Qualification Support Packages (TQSPs) greatly reduce the time and effort of qualifying the LDRA tool suite for use as a verification tool in a project environment. LDRA provides TQSPs for Programming Standards Checking, Structural Coverage Analysis, Data and Control Coupling and Unit Test.
Software testing is an integral part of any software development effort. Software testing generally occurs in phases from unit-level testing through integration testing, system integration testing, acceptance testing, and operational evaluation. Testing should address not only performance-related requirements, but the SSRs as well. The SSS team must interface directly with software developers and the V&V team to ensure that the code correctly implements all SSRs and that the system functions in accordance with the design specifications.
Software systems safety testing includes a variety of test types, many are common testing techniques, others are uncommon for many test programs. The ultimate concern is the effect of software and software errors or failures at the system level. Software itself is not hazardous until interfaced with a hardware system with associated hazards. Therefore, the focus of the software systems safety testing is on the system-level effects.
Safety testing is iterative in nature during all phases and is an integral part of all test processes. Testing at the unit level may result in the addition or modification of tests at the module level. Conversely, anomalies discovered at the module level will likely result in changes to software units requiring retest. These iterations occur at every level of testing through acceptance testing and operational evaluation. However, by the time the system reaches the operational evaluation level, all safety-significant anomalies should be corrected, or the system will receive an unfavorable evaluation.
Section 4.4.2.1 provides guidance on requirements-based tests that focus on verifying that the software implements the high-level requirements in the software requirements specification and software design documents. These tests include safety requirements. The test team will develop test cases and procedures that verify and validate the implementation of the requirements. Section 4.4.2.2 provides guidance on functionality tests which are similar to requirements-based tests but for system level requirements rather than high-level software requirements.
Sections 4.4.2.3 and 4.4.2.4 focus on coverage testing with the goal of ensuring that every possible path through the source code is exercised at least once. Section 4.4.2.5 focuses on stress testing. Sections 4.4.2.8 through 4.4.2.12 provide guidance on a variety of testing techniques that fall into the category of robustness testing.
Section 4.4.2.13 provides guidance on test phases starting with unit testing, through integration of units into modules, and then a complete software system operating on the target hardware platform.
Section 4.4.2.14 provides guidance on regression testing for the software safety team. Regression tests are a subset of other tests run on the system software at various levels of testing. Regression testing ensures that modifications to the software do not adversely affect functionality or safety.
Requirements-based testing typically progresses through five distinct stages:
Following this process will ensure that code has been implemented to address every requirement, that the implemented code has been tested to completeness, and there is no code that is not attributable to a requirement.
It is possible to achieve bidirectional traceability using traditional methods, but automating the process with the TBmanager component of the LDRA tool suite makes it is much less error prone and labor intensive. Using TBmanager, it is easy to see which requirements have been verified, and which have not. During development, the TBrun component of the LDRA tool suite is used to confirm that the functions of a system or program behave as required.
Structural code coverage analysis can be supported by unit test, system test, or a combination of the two, operating in tandem. For instance, a preferred approach might be to use dynamic system test to generate coverage of most of the source code, and to supplement it using unit tests to exercise code constructs which are inaccessible during normal operation.
The LDRA tool suite provides an extensive set of code coverage capabilities including statement coverage, branch/decision coverage, Multiple Condition/Decision Coverage (MC/DC), function and call coverage, data and control coupling coverage, Linear Code Sequence and Jump (LCSAJ) Coverage, and Object Code Verification.
TBjustify manages the documentation of justifications concerning coverage for certification and compliance purposes.
The worst-case execution time (WCET) of a computational task is the maximum length of time that the task could take to execute in a specific environment. Hard real-time systems need to satisfy stringent timing constraints imposed by the nature of the functions they fulfil. Unfortunately, it is not possible in general to calculate definitive upper bounds on execution times for software. Measuring the WCET typically involves stress testing the software and measuring the execution time.
There are further complications in the case of multicore processors. The use of additional cores results in resources being shared between them. Time-related delays occur as users wait for access. The contention that results from threads looking to access Hardware Shared Resources (HSRs) means that an iterative development process that tunes the code and environmental settings to optimize the balance between in different factors is necessary. The TBrun component of the LDRA tool suite together with the TBwcet module enables the optimization of system configuration by means of interference research through measured execution times.
Automatic test case generation can quickly generate a set of test cases that users can review and link to requirements, reducing the time needed to generate a set of requirements-based tests. Additionally, automatic test case generation can be used to create robustness tests – as is the case in applying equivalence class testing and boundary value testing. TBextreme and TBextremePLUS automate the generation of both requirements-based and robustness test cases.
When an embedded system is in the hands of an end user, it will consist of production hardware and software operating in tandem. Many aspects of software behavior are dependent on the hardware underpinning it, and so testing the software in as representative an environment as possible will yield the most reliable results. LDRA tools facilitate target testing by leveraging Target License Packages (TLPs). Each TLP module provides configuration with a specified tool chain and target platform.
Impact analysis can be performed using the TBmanager component of the LDRA tool suite. It is a technique used to determine whether a change or an enhancement to a software system has affected, or has the potential to affect, the existing system. When a change is made and impact analysis is complete, the extent of the re-verification required will be influenced by the number of software modules affected, the criticality of the affected software modules and the nature of the change. Possible decisions are:
The TBrun component of the LDRA tool suite makes re-verification easy to achieve by storing test data for subsequent automated regression test.
Technical note: Achieve DO-178C compliance with LDRA tool suite
Data sheet: LDRA Productivity Package – Aerospace and Defense
Technical briefing: DO-330 test tool qualification for aerospace applications
News article: Ensuring the compliance of avionics software with DO-178C
Webpage: The FACE® Approach and FACE® Conformance
Video: No Magic Teamwork Cloud & LDRA tool suite Bidirectional Traceability
Webinar : MIL-STD-882E and DO-178C Entanglement
Email: info@ldra.com
EMEA: +44 (0)151 649 9300
USA: +1 (855) 855 5372
INDIA: +91 80 4080 8707