According to the Oxford English Dictionary, Artificial Intelligence has existed conceptually since the 1950s. It is an idea that has evolved through the years until today, when current parlance associates it with non-deterministic software, neural nets, and big data – clearly not the case initially!
Today’s Artificial intelligence (AI) is a wide-ranging tool that enables people to rethink how we integrate information, analyse data, and use the resulting insights to improve decision making. AI is impacting software engineering in many ways. From a development aid designed to accelerate developer output to the underlying technology behind machine learning, its potential impact is important today, and potentially pivotal tomorrow.
The Oxford English Dictionary defines Artificial Intelligence (AI) as “The capacity of computers or other machines to exhibit or simulate intelligent behaviour…”. It cites the first reference to AI as being in 1955.
Given the advances in technology since that time and the imprecision of the definition, clearly there will be a range of AI capability starting from the rudimentary and progressing to the cutting edge.
There is a spectrum of AI strength between these extremes, with truly strong, human-like intelligence yet to be achieved.
Again referencing the Oxford English Dictionary, Machine Learning is defined as “The capacity of computers to learn and adapt without following explicit instructions, by using algorithms and statistical models to analyse and infer from patterns in data….” In other words, it’s about creating strong AI systems that can learn and improve from experience.
The data used to facilitate Machine Learning may be classified as “labelled” or “unlabelled”. The term “labelled data” refers to data that has been somehow marked with correct outcomes or target values.
There are four core types of machine learning, each of which involves the training of an algorithm in slightly different ways:
AI/ML presents a challenge to the established order of safety-critical application development. An assurance of reliability, safety, and security traditionally comes as a product of both a cautious approach to change, and traceability to requirements. AI/ML stands as a challenge to both of those notions – and yet offers the promise of systems that are intrinsically safer than those developed more traditionally.
The compromise adopted by the FDA in the United States to date involves exclusively approving AI/ML-based Software as a Medical Device (SaMD) that employs “locked” algorithms – that is, algorithms that yield consistent outcomes in response to consistent inputs. The alternative would be to deploy adaptive AI/ML-based SaMD. These applications would constantly learn, allowing their algorithms to adapt in response to insights gained from real-world usage.
These forms of continuously learning and adaptive AI/ML algorithms might yield outcomes different from their original approved state which contravenes both existing FDA practice and the aims of functional safety standards like IEC 62304.
Referencing AI in its broadest sense, it could be argued that (for example) both automated test vector creation and requirements impact analysis are forms of AI. Both “simulate intelligent behaviour” – after all, these are tests that would take considerable skill and intelligence to perform manually.
Perhaps less disingenuously, it is possible right now to have chatbot applications write articles or develop code on your behalf – bearing in mind that the definition of exactly what that text or code should discuss or do respectively opens a new challenge!
The strong AI typified by Machine Learning is a field of computer science that differs markedly from the traditional development processes – whether they are waterfall, V-model, or CI/CD in nature. It represents a diversification from the traditional requirements/design/develop/test way of doing things. The concept is a new one to us all, and research into how best to embrace the advantages without compromising safety or security is ongoing.
Despite the newness of the term “Artificial Intelligence”, its definition is such that many pre-existing technologies can arguably fall into that category. For examples:
Automated test generation for embedded software using artificial intelligence
The whole point of automated test tools is to streamline the test regimes associated with embedded software test by replacing processes that would traditionally be performed manually. How about the long established “extreme test” facilities provided by LDRA’s TBextreme module? Do they amount to artificial intelligence?
undeniably, the manual creation of tests that exercise code in such a manner would take a lot of human intelligence. On that basis, the TBrun module of the LDRA tool suite is certainly simulating intelligence for this specific task.
Artificial Intelligence and change impact analysis for embedded software
Similarly, requirements traceability can be achieved by manual means. But requirements sometimes change, and tests sometimes fail. In an environment that is both demanding and dynamic in nature, that makes it a project management headache. Simulating the intelligence required to keep track of things helps to alleviate that pain.
Chatbots perhaps present the most accessible form of a stronger, more recognisable AI than that exhibited by existing conventional (albeit sophisticated) software. In principle, leveraging Artificial Intelligence to write source code is a simple exercise. Below is an example of that using the Chat-GPT chatbot. More sophisticated, more focused AI-based code generators are also available.
On first acquaintance, the ease with which that is achieved might be alarming to software developers! However, it is the simplicity of the example problem that makes it appear so trivial. The principle is less easily extended to a highly complex module within a safety-critical control system when the definition of exactly what is required becomes the primary challenge.
At most, then, such an approach will likely be an aid to most developers – not a threat. Indeed, code such as this can be subjected to exactly the same LDRA tool suite tests and checks as any other source code.
Despite the breadth of the dictionary definition, mention Artificial Intelligence to most and it is strong AI that will come to mind. These systems are typically complex, use sophisticated technology such as deep neural nets, are sometimes based on big data, can be poorly specified and can be non-deterministic. All these characteristics create new challenges in terms of functional safety, cybersecurity, and (more generally) validation and verification.
Strong AI is already being implemented in many industrial environments. For example, the automotive sector has not only seen manufacturing robots using AI to manufacture the cars, but also self-driving cars now being widely tested across the world.
Some parts of such systems – software embedded within LiDAR, radar, and camera endpoint devices, for example – will likely be developed using conventional software technology, perhaps in accordance with functional safety or SOTIF standards. The role of such applications should not be underestimated.
However, the decisions taken in response to the received input from these peripheral devices do not leverage software in the same traditional way. In general, that are taken by a system that has been through a “supervised learning” process to ingest large amounts of labelled training data, analysed it for correlations and patterns, and then uses the patterns to make predictions about future states.
This data-centric approach represents a new development regime, with its own set of best practices to be captured by international standards. The following diagrams include references to the committees responsible for the development these standards. For example, the ISO/IEC JTC 1/SC42 committee is responsible for standardization around artificial intelligence.
The adoption of AI (in the modern idiom) and ML is gathering momentum across the safety critical sectors. Some, particularly the automotive sector, are looking to establish best practices and encapsulate them in new standards.
In each case, it is likely that conventional, deterministic programming will have a significant part to play in presenting data in a format that is appropriate for AI algorithms to work with.
A further common challenge is that in each of these sectors, the processes associated with formal functional safety standards simply don’t fit the AI/ML ethos – whether the resulting application is safe, or not.
The notion of self-driving cars needs no introduction, given its newsworthiness in both the technical and general press, and work is ongoing to develop standards for developers of such systems to work to.
The aim is for systems developed in accordance with the best practices defined within these documents to fulfil the objectives of higher level, long-standing, higher-level standards such as ISO 26262 and ISO 21448. They are concerned with functional safety and the safety of the intended function (SOTIF), respectively.
Safety and artificial intelligence in the automotive sector
Two upcoming standards related to ISO 26262 and ISO 21434 are of particular interest in this context.
The opportunities to apply AI/ML in manufacturing are as varied as the industry itself, and so are best illustrated by example such as the Proof of Concept implemented at an Audi manufacturing plant in Neckarsulm, Germany.
Audi assembles up to approximately 1,000 vehicles every day at the Neckarsulm factory, and there are 5,000 welds in each car. To ensure the quality of its welds, Audi has traditionally performed manual quality-control inspections. It is impossible to manually inspect 1,000 cars every day, however, so Audi used the industry’s standard sampling method, pulling one car off the line each day and using ultrasound probes to test the welding spots and record the quality of every spot. Sampling is costly, labour-intensive and error prone. So, the objective of the PoC was to inspect 5,000 welds per car inline and infer the results of each weld within microseconds.
A machine-learning algorithm was created and trained for accuracy by comparing the predictions it generated to actual inspection data that Audi provided. The machine learning model used data generated by the welding controllers, which showed electric voltage and current curves during the welding operation. The data also included other parameters such as configuration of the welds, the types of metal, and the health of the electrodes – much of which was collated using conventional, deterministic software algorithms, likely developed in accordance with IEC 61508 or similar.
The resulting AI models were then deployed with the result was that the systems were able to predict poor welds before they were performed. This substantially raised the bar in terms of quality.
AI/ML and aviation seem unlikely bedfellows. After all, the impressive safety record in that industry has long been founded on a conservative attitude towards new technologies. However, documents such as the EASA guidance for level 1 and 2 machine learning applications highlights that the authorities are fully aware of both the potential for these technologies, and the need to regulate their use.
However, away from the highly safety-critical functionality of fly-by-wire systems and engine control systems, AI/ML applications such as flight path optimisation, predictive maintenance, and flight simulation are already finding favour.
Mirroring the experience of the aviation, AI/ML systems are finding favour in railway operations.
The surge in passenger volumes, intricate route systems, meticulously planned schedules, and the simultaneous presence of both passenger and freight traffic are all contributing factors to an ever-more complex challenge. Concurrently, there’s a growing need for enhanced safety, greater capacity, improved comfort, and heightened sustainability. AL/ML is emerging as a potential aid in finding optimal compromises in fulfilling these conflicting demands.
For more safety-critical applications and away from the logistics of operations, AI/ML is raising regulatory questions. The expectation of improved safety and logistics as a result of AI/ML deployment does not align with the current requirements of the railway regulations for the functional safety of railway systems and their verification procedures. They are not prepared for AI-based applications.
The potential of AI/ML in medical devices has long been recognised, and in 2021 the World Health Organization published its 165-page guidance document Ethics and Governance of Artificial Intelligence for Health.
Medical device developers have used AI/ML algorithms for several years. However, traditional regulatory standards have meant that AI/ML has been used only in the development of the systems. That learning has then been terminated by “locking” the algorithm such that the software became deterministic and could be evaluated through the usual formal certification processes. The FDA has published a list of medical devices that have been approved in this way which includes equipment related to radiology, clinical chemistry, ophthalmic science, and gastroenterology.
The FDA (GUI00020046) and BSI (BS/AAMI 34791:2023) have both recognised that such a position is suboptimal and have released guidance relating to the future deployment of AL/ML in medical devices.
LDRA has been at the forefront of critical application software for almost 50 years, so it will come as no surprise to learn of our involvement with the development of standards that relate to AI.
LDRA’s technical specialists are active participants in the development of three key standards that will establish best practice in this revolutionary new field.
This document will explain those characteristics which are specific to AI-based systems, and the corresponding difficulties of specifying appropriate acceptance criteria.
This document will be an application-specific extension to the standards on SQuaRE. It will describe a process to be applied in the evaluation of quality in AI systems.
This document is an application-specific extension to the standards on SQuaRE. It outlines a quality model for AI systems.
Keeping abreast of developments in this way both presents the opportunity for LDRA to support the community, and for LDRA to ensure that its offerings always remain cutting edge.
Email: info@ldra.com
EMEA: +44 (0)151 649 9300
USA: +1 (855) 855 5372
INDIA: +91 80 4080 8707