The Safety of the Intended Functionality (SOTIF) or ISO 21448:2022 is an international standard adopted by a majority of organizations developing and deploying safe Advanced Driver Assistance Systems (ADAS) and Automated Driving Systems (ADS) at scale. This standard provides a structured approach to managing risks that arise not from systematic and random hardware failures, but from functional insufficiencies and triggering conditions. It’s crucial to note that only the “objectives” in the standard are normative while the rest of the text is useful informative guidance.
When we train automotive and robotics industry professionals on this standard, we observe that they mostly fall into one of two camps – (1) frustrated about the lack of specificity of targets or normative requirements similar to ISO 26262 or (2) relieved that the lack of specificity allows them to define “what is safe enough?” for their organization. Regardless of where they stand, one thing always becomes clear – it’s necessary to carefully formulate defensible criteria for safe deployment of ADAS and ADS, and systematically capture and document such evidence. As we know now, ISO 21448 is not a silver bullet with all the answers – it is a useful tool in our toolbox to help us ask the right questions when developing such complex non-deterministic systems.
In this first installment of our four-part blog series, we will explore the basic concepts of acceptance criteria and validation targets as defined by the SOTIF standard, setting the stage for deeper technical discussions in subsequent blog entries.
Understanding Acceptance Criteria in SOTIF - Qualitative or Quantitative?
At the core of the ISO 21448 risk reduction approach are acceptance criteria – they serve as a litmus test for whether a system’s level of risk is acceptable or not. These criteria are necessary for evaluating the system’s capability to handle real-world driving scenarios without posing an unreasonable amount of risk to vehicle occupants and other road users. Acceptance criteria can be either qualitative or quantitative in nature.
Qualitative Acceptance Criteria: These typically include descriptive measures, focusing on the autonomous system’s behavior in different conditions. For example, ensuring that the responses of a vehicle’s ADAS are intuitive to human operators can be viewed as a qualitative measure. Such a criteria will likely include assessing whether the system provides adequate warnings in potentially hazardous situations or whether its interaction with human users is predictable. The judgement of whether a qualitative acceptance criteria has been sufficiently satisfied is provided by internal and/or external subject matter experts. This can be very challenging and frustratingly subjective.
Quantitative Acceptance Criteria: On the other hand, these involve defining numerical thresholds that must be met to prove the safe performance. These criteria are often defined by specific metrics, such as the maximum allowable rate of unintended lane departures, the minimum acceptable distance a vehicle must maintain from other pertinent obstacles, or the percentage of correctly identified road signs under various environmental conditions. For instance, an acceptance criterion might specify that a lane-keeping assist system should maintain the vehicle within lane markings 99.5% of the time within the specified operational design domain (ODD). These criteria are designed to provide measurable and objective benchmarks that the system must achieve to be deemed sufficiently safe. This is certainly easier said than done. Particularly, the effort involved to quantitatively define the acceptance criteria for major aspects of the dynamic driving task (DDT) can take several person-months of effort. Before we can define what’s quantifiably safe behavior for an ADAS or ADS, we must deeply understand human driving behavior – particularly what does it numerically mean to be a good attentive driver? We have reviewed many US state Department of Motor Vehicles (DMV) driving manuals to only scrape the surface of this challenge.
ISO 21448 emphasizes the importance of these criteria in evaluating residual risk—the risk that remains after all the SOTIF safety measures have been successfully implemented, verified and validated. There are a variety of risk reduction frameworks in safety engineering and standardization that inform these acceptance criteria for reducing or eliminating unreasonable risk, including but not limited to, GAMAB (globalement au moins aussi bon – roughly translated “as a whole at least as good as”), ALARP/ALARA (as low as reasonably practicable/ achievable), MEM (minimum endogenous mortality) and PRB (Positive Risk Balance). We will address these in more detail in the next blog entry.
What are SOTIF Validation Targets?
The next step after acceptance criteria is to define validation targets. These targets are specific, measurable objectives that provide evidence that the system meets the defined acceptance criteria. They are matched pairs. Validation targets are instrumental to the SOTIF verification and validation (V&V) processes, to evaluate whether the system performs safety across its intended operational design domain (ODD) when subjected to “known” and “known” driving scenarios. More on this a little later.
Validation targets can vary depending on the specific functionalities and authorities of the ADAS or ADS. For example, an automated emergency braking (AEB) system might have a validation target specifying that the system must not exceed one (1) false positive AEB engagement over X number of kilometers/ miles/ hours under specific test conditions. These targets ensure that the safe performance of the ADAS or ADS is not only theoretically defined but the implementation is also practically validated through rigorous testing within the ODD.
In ISO 21448, validation targets are closely linked to scenario-based testing — both in the real-world and using high-fidelity simulation environments. This approach involves exposing the system to a wide range of operational scenarios, including those containing corner-case triggering conditions that could challenge its capabilities. By doing so, engineers can hopefully gather sufficient data to prove the system’s overall behavior is satisfying the acceptance criteria defined earlier. The observed truth is that organizations that empower their safety teams and continue to invest the time and effort to systematically specify and iterate on their acceptance criteria and validation targets continue to endure. This is the real secret sauce when it comes to highly automated vehicle development – this information is seldom shared publicly.
How do we define Acceptance Criteria and Validation Targets for Known (Area 2) and Unknown (Area 3) Hazardous Scenarios?
In our most recent training, a colleague developing SAE Level 4 Autonomous Trucks rightly challenged us with this question – “Wouldn’t you need different approaches for known and unknown scenarios while defining acceptance criteria and validation targets?”. For context: In ISO 21448, scenario areas 2 and 3 represent “known hazardous” and “unknown hazardous” scenarios, respectively. Each area, therefore, requires a distinct approach to defining acceptance criteria and validation targets.
For SOTIF Area 2 (known hazardous scenarios), acceptance criteria are well-defined because the scenarios and potential hazards are already identified. The criteria focus on ensuring that the system can handle these scenarios without causing harm. Validation targets for area 2 involve specific tests that replicate these known hazardous conditions to verify that the system’s response meets the defined acceptance criteria.
For SOTIF Area 3 (unknown hazardous scenarios), the challenge lies in the unpredictability and the lack of knowledge of the triggering conditions. Therefore, ISO 21448 suggests a more exploratory approach to validation. These validation efforts often include extensive real-world and simulation-based testing strategies, including constrained random testing and statistical analyses to uncover and assess these unknown scenarios. The acceptance criteria in Area 3 might not be as explicitly defined due to the nature of the unknown scenarios, focusing instead on reducing the likelihood of hazardous outcomes through robust system design and broad scenario coverage.
Summary
In this first part of our blog series, we’ve introduced the basic concepts of acceptance criteria and validation targets within the context of ISO 21448. These elements are critical for proving that ADAS and ADS are operating safely and effectively in real-world conditions. In the next installment, we will delve into the building blocks and introduce mathematical frameworks that underpin these criteria and targets. Stay tuned as we continue to break down the concepts within the SOTIF standard and hopefully make it less intimidating.