Scenario Testing¶
Test protocol — instantiates Model-Based Regulation
Checks the regulator against a curated set of plausible, extreme, and boundary situations, asking of each: does it stay within safe limits and degrade gracefully?
Scenario Testing checks the regulator against a deliberately chosen set of situations — the plausible, the extreme, and the awkward boundary cases — and asks a pass/fail question of each: in this situation, does the policy keep the system within its safe limits, and when pushed past them, does it fail gracefully into a known safe state? Where the other tests sweep ranges or replay history, scenario testing is curated and discrete: a hand-built portfolio of named cases, each with an acceptance criterion decided in advance. Its purpose is not to estimate typical performance but to confront the policy with the specific situations someone is worried about and get a clear verdict on each.
Example¶
A reservoir's spillway gates are governed by an automated policy balancing flood risk downstream against water storage. Before a revised policy is trusted, engineers assemble a scenario portfolio: a routine spring melt (plausible), a probable-maximum-flood inflow arriving in a single night (extreme), a flood that crests exactly as a gate actuator jams (boundary/compound), and a sensor blackout during rising water (degraded-information). Each scenario carries a pre-agreed acceptance criterion — downstream flow must stay under the levee-design limit; storage must not exceed the dam's safety pool.
Run through the portfolio, the revised policy passes the routine and extreme-inflow cases but fails the jammed-gate scenario: it keeps commanding the stuck gate and lets the reservoir climb past its safety bound instead of dumping through the remaining gates. It also fails to drop into the pre-defined safe holding configuration when the sensors go dark. Those two red cells are the deliverable — not a score, but a precise list of the situations the policy does not yet survive — and they send it back for a fix aimed exactly there.
How it works¶
Scenario testing is defined by how its cases are chosen and judged, not by any engine that runs them:
- Cases are selected for worry, not coverage. The portfolio deliberately spans plausible, extreme, and boundary/compound situations — especially the compound ones, two things going wrong at once, that ranges and averages miss.
- Each case has a verdict decided in advance. Acceptance criteria — a safety limit not to breach, a fallback to reach — are fixed before the run, so the result is an unambiguous pass or fail rather than a number to interpret.
- Failure into the fallback is itself a test. It checks not only that the policy stays safe when it can, but that it degrades into a known safe state when it cannot — surviving the situation, not just handling the happy path.
- It borrows an execution environment. The scenarios are usually run on a simulator or digital twin; scenario testing supplies the cases and criteria, not the dynamics.
Tuning parameters¶
- Scenario portfolio — which situations make the list and how extreme they go. A richer portfolio catches more but costs authoring effort, and can never be exhaustive.
- Acceptance strictness — how hard the pass bar is per scenario. Strict bars catch subtle failures but can red-flag behaviour that is actually tolerable.
- Compound depth — how many things go wrong at once in a scenario. Deeper compounds expose brittle interactions but explode the number of cases.
- Plausible-to-extreme balance — how the portfolio is weighted between everyday and once-in-a-lifetime cases. Over-weighting extremes hardens against black swans but may neglect common failure modes.
- Fallback-reach requirement — whether merely "staying safe" passes, or the policy must also reach the defined safe state. Encodes how much you trust graceful degradation versus prevention.
When it helps, and when it misleads¶
Its strength is that it gives a clear, per-situation verdict on exactly the cases stakeholders fear, and it is the test that most directly checks the two things regulation must never get wrong — breaching a safety bound and failing to reach a safe fallback.
Scenario testing can only ever cover the scenarios someone thought to write. It is blind to the situation nobody imagined, so a wall of green can breed false assurance against the case that actually arrives — the beyond-design-basis event[1]. It also invites teaching to the test: tuning the policy to pass the named scenarios while leaving the space between them fragile. The discipline is to treat the portfolio as a floor and not a ceiling, refresh it with every near-miss and post-incident finding, and pair it with range-sweeping and adversarial methods that probe the gaps a curated list leaves.
How it implements the components¶
Scenario Testing fills the safety-acceptance slice — the components that decide whether the regulator behaves acceptably in the situations that matter most:
safety_bound— each scenario operationalizes a safety limit into a concrete pass/fail check the policy must not breach.safe_fallback_state— boundary and degraded scenarios verify that, when it cannot stay safe by control, the policy reaches the defined safe state rather than pressing on.
It supplies cases and criteria, not the machinery around them: the execution environment comes from Digital Twin Trial, the parameter-range robustness from Sensitivity Analysis, and the recorded history from Historical Replay.
Related¶
- Instantiates: Model-Based Regulation — supplies the per-situation safety verdicts before live regulation.
- Consumes: Digital Twin Trial usually provides the environment the scenarios are executed in.
- Sibling mechanisms: Digital Twin Trial · Sensitivity Analysis · Historical Replay · Model-Failure Red Team · Shadow-Mode Evaluation
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Scenario Testing operates as an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation because it checks the regulator against a curated set of plausible, extreme, and boundary situations, asking of each: does it stay within safe limits and degrade gracefully?.
Independent corroboration: The frozen evidence defines Scenario Testing as 'Checks the regulator against a curated set of plausible, extreme, and boundary situations, asking of each: does it stay within safe limits and degrade gracefully?', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Engineering & Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Verifying a regulator against plausible, boundary, and extreme situations for safe limits and graceful degradation is engineering test design. NASA product verification uses models, simulations, hardware-in-the-loop, and relevant-environment testing to provide objective evidence; systems cybernetics supplies regulator theory.
Related originating lineages:
- Computer Science & Software Engineering — Automated scenario testing independently checks graceful degradation.
- Operations Research — Robust-control analysis materially evaluates safe-limit compliance.
- Statistics & Experimental Design — statistics_experimental_design contributes sampling, sensitivity, inference, and controlled testing to the mechanism's formative or independently convergent form; that contribution does not displace the primary engineering_design lineage.
- Systems Thinking & Cybernetics — Scenario Testing's terminology and operating form—checks the regulator against a curated set of plausible, extreme, and boundary situations, asking of each: does it stay within safe limits and degrade gracefully?—are rooted most directly in systems thinking, feedback control, and cybernetics.
Review resolution: The blind reviewers disagreed on primary lineage (engineering_design versus systems_cybernetics); authoritative or primary research supports engineering_design as the best historical origin. Verifying a regulator against plausible, boundary, and extreme situations for safe limits and graceful degradation is engineering test design. NASA product verification uses models, simulations, hardware-in-the-loop, and relevant-environment testing to provide objective evidence; systems cybernetics supplies regulator theory. The cited NASA, Systems Engineering Simulator; NASA Systems Engineering Handbook, Product Realization directly supports the defining operation used in that choice. All independently supported contributing domains are retained without an arbitrary cap, while domain_reach=multi_domain records later applicability separately from provenance.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
Notes¶
Scenario Testing and a Model-Failure Red Team pull in opposite directions and need each other. Scenario testing confirms the policy survives the situations you listed; a red team's job is to invent the situation you didn't. A green scenario suite tells you the known hazards are covered — not that the unknown ones are — so the two together, rather than either alone, is what earns confidence.
References¶
[1] International Atomic Energy Agency. IAEA Safety Glossary: Terminology Used in Nuclear Safety and Radiation Protection, 2018 Edition. STI/PUB/1830. Vienna: IAEA (2019). Defines beyond-design-basis accident conditions as more severe than, or not encompassed by, the design basis. registry ↩