From UCAs to Test Scenarios: Building a Coherent Validation Pipeline for AVs
This article was written by an SRES consultant and systems engineer specializing in autonomous-vehicle safety, safety analysis, and safety-critical system evaluation. It examines how Systems-Theoretic Process Analysis (STPA), Unsafe Control Actions (UCAs), operational design domains, and the ISO 3450x scenario standards can be connected to develop a traceable validation pipeline for automated driving systems.
Looking to go deeper? Explore SRES’s Safety Analyses for Automated Driving Systems (ADS) Training and Workshop for practical work with FMEA, FTA, Cause Tree Analysis, and STPA in the context of ISO 26262 and ISO 21448. SRES also provides Autonomous Product Development consulting and broader automotive safety consulting to support safety analysis, validation strategy, and the responsible development of autonomous systems.
Introduction
Proving Autonomous Vehicles (AVs) behave in an acceptably safe manner is far harder than building them, and this is a challenge that every AV developer goes through. Let’s consider the verification and validation effort for AVs – it is practically impossible to drive the AV around test tracks or public roads and hope to encounter edge cases or conditions that trigger insufficiencies in the AV. The more practical approach would be to analyze the system, identify the weak points, and generate a test catalog that is both systematic enough to trust and finite enough to execute. This post walks through one disciplined approach to tackle the issue, and the way a series of international standards fits together to support it.
Why start with UCAs rather than scenarios?
Brainstorming test scenarios is like falling into a rabbit hole: a pedestrian steps out, a lead vehicle brakes hard, fog rolls in, and the list goes on, and you never know what you have missed, because the list was never derived from anything.
To solve this, engineers start from a safety analysis perspective, more specifically an analysis that identifies the potential hazards of unsafe outputs from the AV. A systems-theoretic process analysis (STPA) is one such process that models the AV as a set of control actions — accelerate, brake, steer, maintain lane — issued by a controller. The STPA process begins by defining losses, and this includes outcomes that are to be avoided (collision, injury to road users, damage to property). Then, the system-level hazards that lead to the defined losses are identified, along with the system-level safety constraints that help prevent the hazards leading to losses. A hazard is a system-level state, not a control action.
Once the control structure is modeled, the analysis derives the ways a control action becomes unsafe in the context of a hazard: provided when it should not be, not provided when it should be, provided too early or too late, or stopped too soon. These are called Unsafe Control Actions (UCA), and they feed into the scenario derivation process. Other safety analysis techniques exist such as functional insufficiency to triggering conditions (FI2TC), or TC2FI that traces the relationship between system insufficiencies and triggering conditions that lead to the system behaving in an unsafe manner. Cause Tree Analysis (CTA), a top-down approach to deduce the insufficiencies and triggering conditions that cause hazardous events. Decomposing a hazardous event using the CTA process yields root causes that can be traced back to specific triggering conditions and operational situations. Although the developer can take any of the above mentioned techniques to study the system, this blog post will focus on deriving test scenarios through STPA.
In this process, instead of imagining scenarios and hoping they cover the hazards, you enumerate the hazards first and then ask what situations would potentially expose each one. Every test scenario can then be traced back to a specific way the system could behave unsafely.
An example of a control action that is commonly seen while performing this analysis for AVs is “command automated emergency braking”. And, for this control action, a UCA might be something like:
“the ADS provides emergency braking when it is not needed”.
It is to be noted here that the UCA statement identifies a controller, a control action, and the unsafe behavior by the controller, but at the same time, the statement does not describe the world conditions, or what caused the controller to command an unsafe maneuver.
How does a UCA become a concrete test case?
The process of translating UCAs to various abstractions of scenarios is a process of progressive refinement, and it happens in stages. A UCA on its own does not describe a situation; it only identifies an unsafe control action. It becomes a class of situations only once it is paired with a causal scenario and placed in a use-case environment. Consider the above-mentioned example, an unnecessary emergency braking command is completely inconsequential on an open road with no one behind. The same unnecessary hard stop becomes dangerous only when there is a vehicle following close behind, or a driver not expecting the ego to brake on a green light. Following the identification of the UCAs, the next step in the STPA process is to identify the causal scenarios that could cause the UCA, and these could include perception limitations/errors, controller logic flaws, or actuator delays. The combination of the UCAs with the causal scenarios, in a given environment creates the path to hazards and losses.
Take the unnecessary-braking UCA in the above example and pair it with a plausible causal scenario, say, the perception system misclassifies a stationary object off to the side (such as a vehicle stopped in an adjacent turn lane), as an in-path obstacle, and the planner treats it as an imminent collision threat. Now the developer can begin to describe the situations in which this combination produces a hazard. That description, written in natural language forms the functional scenarios: “the ego proceeding through a signalized intersection on a green light, a vehicle following close behind, a non-threatening object present just off the ego’s path, and the perception system intermittently flagging it as in-path”.
When making the description more formalized and descriptive, it forms the abstract scenarios. The abstract scenario fixes what can vary and how the entities relate, without yet committing to how much anything varies. It is the bridge between the prose of the functional level and the quantified space of the logical level.
Upon adding actors and parameters (value range), we can create logical scenarios. A logical scenario contains parameters like speeds, distances, timings, environmental conditions, and each parameter has a range. Each named quantity from the abstract scenario is given bounds, such as the ego’s approach speed might range from 40 to 60 km/h, the following vehicle’s gap from 10 to 25 metres, the lateral offset of the off-path object from 2 to 4 metres, the commanded deceleration from 0.4 to 0.8g, etc.
The final step is instantiation through the ranges of value defined at the logical level, and choosing specific values within those ranges produces a concrete scenario. For example, a concrete scenario can look something like:
“ego enters the intersection at 50 km/h on a green light, a following vehicle 18 metres back at the same speed, a delivery truck stopped in the right-turn pocket 2.5 metres laterally off the ego’s path, with the system triggering a 0.6g deceleration despite no real in-path threat, on dry asphalt at midday.”
So technically, one logical scenario can therefore yield many concrete scenarios, each probing a different point in the parameter space. The art lies in sampling that space well: covering the meaningful variation without drowning in redundant near-duplicates.
How do the ISO 3450x standards interlock in practice?
Managing this parameter space effectively requires a rigorous methodology to ensure consistency, which is exactly where the broader ISO 3450x series becomes essential. The scenario-related standards in the ISO 3450x series are best understood as a connected toolkit rather than a list of separate references. The ISO 3450x series provides a standardized methodology for creating, managing, and executing test scenarios for automated vehicles. There are five standards in this series:
- ISO 34501: Vocabulary
- ISO 34502: Scenario-based safety evaluation framework
- ISO 34503: Specification for operational design domain (ODD)
- ISO 34504: Scenario categorization
- ISO 34505: Scenario evaluation and test case generation
ISO 34501 establishes a unified, and precise vocabulary that eliminates engineering ambiguity for AV stakeholders. This acts as a foundational baseline to implement the rest of the ISO 3450x series.
ISO 34502 provides a framework for identifying and generating scenarios for testing and validating the performance of the ADS. By categorizing real-world risks across traffic, perception, and control, it provides the core methodology to actually execute your SOTIF strategy. The UCA from the example above perfectly fits into the perception-related failure category, where a stationary roadside object misclassified as an in-path obstacle is exactly the kind of perception risk factor 34502 is designed to identify. The framework is what tells the developer the unnecessary-braking hazard is worth analyzing in the first place, and it pairs naturally with the STPA-style UCA + causal scenario reasoning we used to derive it.
ISO 34503 provides a hierarchical taxonomy to enable a standardized definition of operating conditions that cover environmental, geographical, and temporal conditions under which an ADS is designed to operate. At the top level, it categorizes the ODD into scenery elements, environmental conditions, and dynamic elements. In the above-mentioned example, the developer can judge whether “a signalized intersection with a stopped vehicle on the shoulder” is a situation that the system should handle, and must be tested, based on whether this situation is within the ODD specification. If yes, then the scenario is a valid and necessary test for the control action, but if it is out of the domain, it can be a boundary case rather than a nominal one.
ISO 34504 provides the taxonomy for categorizing and tagging scenarios. It defines a controlled vocabulary for actors, maneuvers, road geometry, and the relationships between road users. This taxonomy is the connective tissue that lets the scenario in the above-mentioned example be tagged as “following vehicle, same lane, similar speed” and “stationary object, adjacent turn lane” a way that every engineer and tool can read and understand.
Finally, ISO 34505 provides the framework to evaluate driving scenarios and systematically expand them into executable test cases. It details the exact parameters each test case requires, spanning unified identifiers, test objectives, execution platforms, and expected results.
The interdependency runs in both directions. The ODD definition bounds the scenario space by restricting the developer’s imagination in designing test scenarios. The scenario taxonomy provides the vocabulary to populate that space precisely, and the UCA together with its causal scenario defines which regions of that space are worth probing, as those are the regions where unsafe behavior could emerge. No single standard delivers a test catalog. The catalog is what emerges when they are used together.
What does a well-structured scenario catalog actually give the developer?
When the process is followed end to end, the output is more than a pile of test cases. It is a traceable safety argument. Each concrete scenario links upward to the logical scenario it was sampled from. Each logical scenario links to the UCA that motivated it. Each UCA links to the hazard analysis of the system’s control structure. And every scenario is described in vocabulary that can be checked against the ODD. The result is a chain of reasoning that runs from “here is a way the system could be unsafe” all the way down to “here is the specific, executable test that demonstrates we considered it.”
This traceability is what turns a test catalog into evidence. A regulator, an assessor, or an internal safety reviewer can pick any test case and walk the chain back to the hazard it addresses — or pick any hazard and confirm that it has been exercised. Coverage becomes auditable. Gaps become visible. The catalog is no longer a static artifact but a living structure that can grow as the ODD expands or as new hazards are identified.
Why does keeping the scenario space physically plausible matter?
It is a waste of resources and labor to design scenarios that are physically impossible to test, and even worse, it can misguide coverage metrics by inflating the catalog with scenarios that are not useful. It is very important to assign plausibility constraints while developing test cases, and these are rules that encode what the physical world permits.
This is also where having a good understanding of the ODD definition is crucial. When the ODD is thoroughly studied, combinations of ODD elements that don’t make sense for the level of maturity of the AV under test, can be categorized as out of ODD, and avoided. This approach helps in narrowing the ODD to focus effort on the in-domain, physically plausible, hazard-relevant scenarios, while still being able to reason explicitly about the boundaries.
Where does this leave the developer?
The central lesson is that safe-by-design validation of AVs is not achieved by any single technique or any single standard. It is achieved by a disciplined chain: hazard analysis identifies the UCAs; those actions motivate logical scenarios; the ODD bounds where those scenarios may lie; a shared taxonomy lets the whole thing be described consistently; and systematic sampling, constrained by physical plausibility, turns the logical scenarios into a finite, executable, traceable catalog.
The engineering value lies in understanding the seams between them — knowing which standard answers which question, and how the answers compose. A test catalog built this way is not just a set of things to run. It is an argument, and a defensible one, that the system has been confronted with the situations that matter.
Although the STPA process helps in identifying UCAs, it does not by itself guarantee complete coverage, as it depends on the analyst and the thoroughness of modeling the control structure. Additionally, the STPA process also does not help in quantifying or prioritizing UCA, i.e., it does not specify how likely or severe a hazard is from another. To make those judgments, additional risk assessment methods are incorporated. The process described in this blog can be considered as an approach to trace scenarios to UCAs, and not a proof of coverage or completeness.
Have insights or questions? Send us an email at info@sres.ai or leave a comment below—we welcome thoughtful discussion from our technical community.
Interested in learning more about our approach? Explore why teams choose SRES training and how we support organizations with consulting for Automotive and Physical AI applications.
