Like open-source software a few years ago, artificial intelligence (AI) and machine learning (ML) are the newest frontiers for safety-critical embedded software. These technologies have triggered a rethink of how we, as humans, integrate information, analyze data, and make decisions. As practitioners in safety-critical software, we must understand how to accommodate the huge promise of AI/ML without compromising functional safety processes and certification.

Defining artificial intelligence and machine learning

Given the noise and uncertainty around AI/ML, fueled primarily by generative AI techniques, it is useful to turn to a broader definition of terms that encompass all potential capabilities brought to safety-critical software development.

The OED defines AI as “The capacity of computers or other machines to exhibit or simulate intelligent behaviour; the field of study concerned with this.” It also specifies that more recent uses of the term refer to “The capacity of computers or other machines to exhibit or simulate intelligent behaviour; the field of study concerned with this.”G

There are two major classifications of AI:

• Narrow or weak AI that excels at the task it has been trained for but lacks general human-like intelligence.
• General or strong AI that possesses true human-like intelligence.

The OED defines machine learning as “The capacity of computers to learn and adapt without following explicit instructions, by using algorithms and statistical models to analyze and infer from patterns in data.” In other words, ML is about creating strong AI systems that can learn and improve from experience.

One could argue that conventionally programmed and deterministic software applications – in use for decades – fall under the classification of weak AI. Consider an aircraft’s flight management system or an automotive ECU. They are built to perform specific tasks in a deterministic manner and generally do them very well.

Similarly, it could be argued that automated test vector creation is a form of weak AI because it “simulates intelligent behaviour” by deriving unit tests from existing code. As illustrated in Figure 1, developing and performing these tests manually would require considerable skill and intelligence. Yet automated test tools perform them comprehensively and efficiently.

Figure 1. Automatically generated test vectors inside the LDRA tool suite (source: LDRA).

AI-assisted software development tools present an obvious form of a stronger AI than that exhibited by conventional applications. Assistants such as Microsoft Copilot, Amazon CodeWhisperer, and Google Gemini Code Assist all offer ways to ease development tasks traditionally performed by humans. Figure 2 shows an example of a coding prompt and answer using the ChatGPT chatbot.Figure 2. Using ChatGPT to answer a basic coding question (Source: ChatGPT)

While the ease with which code can be generated may raise suspicion as to its usefulness, the use of AI programming tools will only continue to grow as language models and techniques improve.

Strong AI presents a bigger challenge to the established order of safety-critical applications. Traditional assurances of reliability, safety, and security come from a cautious approach to change and rigorous traceability to requirements. Strong AI challenges these notions with speed and a somewhat opaque view into the inner workings of training data and language models. It also offers the promise of advanced embedded systems that are intrinsically safer than those developed more traditionally.

Example: AI in medical device software

The U.S. FDA has been proactive in the adoption of AI/ML-based Software as a Medical Device (SaMD) that employs “locked” algorithms – those that yield consistent outcomes in response to consistent inputs. Less clear is how the development of adaptive algorithms – those that respond to and potentially change output based on learnings – should be handled. This lag has placed manufacturers in a holding pattern between AI/ML innovation and potentially contravening FDA guidelines, such as IEC 62304, with applications that may yield outcomes different from their original approved state.

A deeper look into medical device regulations yields promise. The International Medical Device Regulators Forum (IMDRF) SaMD risk categorization framework provides a systematic approach to classifying the risks of software intended for medical purposes. It helps regulatory authorities, manufacturers, and other stakeholders determine the appropriate regulatory requirements and ensure that developed software meets the necessary safety and effectiveness standards.

According to this framework, the intended use of any SaMD, whether AI/ML-based or not, can be classified according to a spectrum of impact on patients, as illustrated in Figure 3. Here, “I” represents the lowest risk, and “IV” is the highest. This impact is based on factors such as the intended use of the software, the significance of the information provided by the software for making medical decisions, and the potential consequences of software failure.

Figure 3. Software as a Medical Device impact on patients per the International Medical Device Regulator’s Forum SaMD risk categorization framework (Source: IMDRF)

The IMDRF framework helps assign SaMD into different risk classes or categories that guide the level of regulatory scrutiny and requirements applied to the software. As this classification is agnostic of the methodology used to create the software, it can be applied to SaMD dependent on AI/ML-based techniques.

Mitigating AI/ML risks in safety-critical applications

There are few, if any, safety-critical systems based entirely on AI/ML. For the most part, the outputs of AI/ML-based software components are integrated into conventionally engineered applications, whether in the same system or communicated to another device.

Therefore, it is logical to extend the principles of domain separation promoted by IEC 62304. This approach contains AI/ML algorithms within dedicated domains and enables domains that receive AI/ML-generated data to mitigate any risks associated with it.

As illustrated in Figure 4, existing taint analysis tools can test and validate data flows coming from AI/ML components. Using such tools, developers of consuming software items know the data ranges that make sense and can design software to “sanity check” incoming and outgoing data—whether from an AI/ML domain or elsewhere.

Figure 4. Taint analysis using the LDRA tool suite. (Source: LDRA)

Adapt now before getting left behind

AI/ML-based systems are here. They’re used to write code and perform innovative functions onboard safety-critical platforms today.

The fact that most, if not all, of these components operate inside conventionally developed software means there are opportunities for risk mitigation, either in the form of conventionally developed, deterministic software or with humans in the loop. Existing best practices, particularly those relating to risk-analysis approaches and domain separation, can be leveraged to good advantage.

Read here – The impact of AI/ML on qualifying safety-critical software – Embedded.com