Comparing ISO/IEC TR 5469 and ISO/IEC TS 22440
This article compares ISO/IEC TR 5469 and ISO/IEC TS 22440, outlining how the field is moving from foundational guidance on AI-related functional-safety risks toward more actionable engineering requirements for AI safety-related control systems. It was authored by an SRES consultant and systems engineer specializing in autonomous-vehicle safety, safety analysis, and safety-critical system evaluation, with deep experience developing safety-case frameworks and simulation-based validation approaches for autonomous systems.
Looking to go deeper? Join SRES for Physical AI Safety Week, including IEC 61508 Functional Safety (IFSP) Training and ISO/IEC TS 22440 Functional Safety and AI Systems Training. Need more than training? Explore our Physical AI consulting services, including Robotics Safety, to support the development of safe, secure, and compliant robotic systems.
Introduction
The last four years have witnessed an enormous growth in the use of Artificial Intelligence (AI) in various domains. The speed with which technology is growing is not only rapid, and is changing the way we think, reason, and create. Just a few years ago, AI could be seen working behind the scenes performing supporting tasks like powering the algorithm behind the Netflix recommendations, social media feeds or Email spam filters, and so on. Over the past decade, we have also seen AI (particularly machine learning) deployed across domains, from industrial robots to perceiving, planning, and controlling Autonomous Vehicles (AV).
This paradigm shift is both exciting and concerning. It is exciting as we get to witness the dawn of the AI era and the capabilities of the current AI models, whether in generating ultra-realistic videos or controlling humanoids that perform chores around the household. The concerns stem from the safety-related issues that come along with these technologies. Unlike traditional software, where the software is programmed to perform in a specific/ deterministic manner giving the developer and the user a better understanding of how and when the software can fail, AI does not behave deterministically.
AI is primarily developed by training models on large amounts of data. This data is representative of what the AI model is expected to encounter as inputs in the real world upon deployment. As the model learns from the data it is trained on, it develops internal representations of what each object is and improves its confidence in detecting those objects. Upon deployment in production, when it encounters an object, it compares the parameters (such as shape, size, color) of the object with the knowledge it gained through training and provides a prediction of what that object could be.
With that being said, the predictions made by the model can be incorrect or inconsistent due to noise, distribution shift, or adversarial manipulation. This is a safety-critical problem when AI models are deployed on robots or autonomous vehicles, where the function of the model is to detect a pedestrian or a stop sign in time, but when it fails to do so, it can lead to collisions causing serious injuries or even fatalities. To address these challenges, the automotive industry has published the state-of-the-art standard ISO/PAS 8800, which guides developers on responsibly creating and validating models that are “safe enough” for deployment into the real world.
Bridging AI and Functional Safety
Functional safety deals with the traditional/ known failure modes of software and defines safety mechanisms to detect, mitigate, or reduce the impact of these failures. With the rise in the use of AI models alongside other hardware and software components in robots and other physical AI systems, standardization committees are working to bridge this gap for better understanding the source of AI errors and implementing appropriate mitigation and control measures similar to what we have in functional safety.
One such effort is the Technical Report “ISO/IEC TR 5469 Artificial intelligence — Functional safety and AI systems”, published in 2024. Being a technical report, ISO/IEC TR 5469 provides guidance and context. There are no normative requirements with “shall”s. This serves as a stopgap measure to provide guidance to the industry while the formal normative standard is developed (which can take years) The replacement for ISO/IEC TR 5469 will be “ISO/IEC TS 22440 Artificial intelligence — Functional safety and AI systems” which will provide normative requirements and informative guidance for designing, developing and deploying AI Safety-related Control Systems (AI-SCS). While ISO/IEC TR 5469 lays the theoretical groundwork and a shared vocabulary, the upcoming ISO/IEC TS 22440 expands into actionable, verifiable engineering requirements and work products.
ISO/IEC TR 5469: Foundational Concepts
Unlike normative standards that dictate strict, mandatory rules, ISO/IEC TR 5469 is a technical report, it serves as an informative, guidance-based document describing properties of AI, related risk factors, and high-level methods for using AI inside (or alongside) safety-related functions. It acts as the “bridge” between the AI community and the functional safety community. The scope of ISO/IEC 5469 covers describing the properties, related risk factors, available methods, and processes related to:
1) use of AI inside a safety-related function to realize the functionality;
2) use of non-AI safety-related functions to ensure safety for an AI-controlled equipment;
3) use of AI systems to design and develop safety-related functions.
The standard introduces a three-stage realization principle for incorporating AI into safety-related systems when compliance with existing functional safety standards cannot be demonstrated directly. The three-stage realization principle consists of:
- Data acquisition: The initial phase involves defining the input tasks and the collection of data used to train the AI system. The safety concerns at this stage stem from poor data quality, noise, and inherent bias. If the model is trained on flawed data, its performance will be compromised.
- Knowledge induction from data and human knowledge: Once sufficient data is collected, the model is trained, where it learns patterns from the labeled training data. Risks at this phase include overfitting, where the model memorizes the training data but fails to understand the broader concept, and stochasticity, where repeated training runs yield slightly different results due to random initialization.
- Processing and generation of outputs: This is the operational inference phase, where the trained AI model processes real-world inputs to generate an output or decision. At this stage, probable safety concerns include distribution shifts, where the real-world data looks different from the training data, and intentional adversarial attacks that alter what the perception system sees.
Although ISO/IEC TR 5469 lays the essential groundwork by fostering awareness of the unique risk factors introduced by AI, it stops short of providing actionable requirements for developers. To transition from theoretical comprehension to practical application, developers need to adopt ISO/IEC TS 22440. This standard acts as the normative bridge, translating AI safety concepts into engineering requirements for designing, testing, and continuously monitoring the AI Safety-related Control System (AI-SCS), and ultimately aligning these data-driven concepts with established functional safety standards such as IEC 61508, and ISO 26262.
ISO/IEC TS 22440: Operational Requirements
ISO/IEC TS 22440 is a three-part series standard, with the first part providing requirements for classifying the AI SW component, and performing relevant lifecycle activities including performing a HARA, AI fault analysis, strategies to mitigate the identified faults, verifying and validating the performance of the AI-SCS.
An AI software (SW) component is classified based on whether it is part of the AI-SCS. If it is, the classification depends on whether the component’s failure can cause a hazardous output within the AI-SCS. If it is not part of the AI-SCS, but rather a tool used to develop the SCS, it is evaluated based on whether errors in the AI SW tool can propagate to the AI-SCS and cause hazardous outputs. Additionally, the standard categorizes the AI SW component based on the methodology used to build it, specifically looking at its overall traceability and the extent of ML involvement.
The information provided by ISO/IEC TS 22440 to support the HARA process is used as a complement to traditional functional safety HARAs. The process comprehensively identifies all potential hazards, failure modes, and event sequences across the ODD, including faults and foreseeable misuse. Following the HARA, the standard classifies the AI faults into categories related to the creation of the model, inputs, pre-processing errors, insufficiencies in the AI models, and the AI monitor. With the faults identified, the standard guides developers to pick architectural elements to mitigate the faults, and validate the performance of those elements in detecting and mitigating the faults. As data is key in developing an AI model, the standard emphasizes the importance of the quality of the data used to train and develop the AI model.
Overall, this emerging standard at the intersection of AI and Functional Safety aims to provide a comprehensive framework for developers to create, train, and deploy safe AI models for use in robotics and autonomous vehicles.
Conclusion
In summary ISO/IEC TR 5469 acts as an educational manual dedicated to illuminating the theoretical vulnerabilities and developmental phases of AI, while ISO/IEC TS 22440 details a stringent, executable blueprint for real-world implementation. ISO/IEC TS 22440 shifts the focus from abstract comprehension to applied engineering by imposing rigorous categorization protocols, exhaustive vulnerability checks, and specific architectural measures required to maintain safe operating conditions in automated machinery.
Have insights or questions? Send us an email at info@sres.ai or leave a comment below—we welcome thoughtful discussion from our technical community.
Interested in learning more about our approach? Explore why teams choose SRES training and how we support organizations with consulting for automotive and physical AI applications.
