Sensitivity Check¶
Analysis method — instantiates Simplification Audit
Varies the variables a simplification fixed or dropped to see whether the decision it supports actually changes — separating omissions that are harmless from ones that are decision-critical.
Sensitivity Check takes each thing a simplification assumed away — a variable held constant, a term rounded off, a factor left out — and puts it back and moves it, to see whether the artifact's answer moves with it. Its defining idea is that relevance is measured, not asserted: an omission matters exactly to the degree that restoring and perturbing it changes the conclusion the simplified artifact is used to reach. Everything else in the audit debates whether a dropped detail could matter; this mechanism turns the debate into a swing. If varying a suppressed variable across its plausible range never flips the decision, the simplification is vindicated for that variable; if a modest perturbation flips it, the omission is decision-critical and the audit now knows precisely which one and by how much. The output is a ranked map of omissions by how much leverage each has over the outcome.
Example¶
An airline's revenue-management team uses a simplified fare rule that sets each route's price from just two inputs — days-to-departure and current load factor — dropping the messier variable of competitor pricing, on the theory that competitors mostly track each other. Before trusting the rule on a newly contested transcontinental route, an analyst runs a Sensitivity Check. They reintroduce competitor price as a variable and sweep it across a plausible band — say, illustratively, competitors sitting anywhere from 15% below to 15% above the rule's fare — and watch what the rule's recommended price and the resulting expected revenue do. On thin, monopoly-like routes the recommendation barely twitches: the omission is harmless there. On the contested route, a competitor undercut of roughly 10% swings expected revenue enough to reverse whether the rule's price is profit-maximizing at all. The check reports this as a ranking: on contested routes, competitor price is the dominant omitted driver; on monopoly routes it is inert. That single finding tells the audit exactly where the two-input simplification is safe and where it must be widened.
How it works¶
- Pick the perturbation set. Restore the specific variables and assumptions the simplification suppressed, one at a time and in the combinations most likely to interact.
- Sweep across plausible ranges. Move each restored variable over the span it could realistically take, not a token nudge, so the test probes the decision's actual exposure.
- Read the decision, not the number. Track whether the conclusion the artifact supports changes — the chosen action, the ranking, the pass/fail — rather than whether an intermediate output wiggles.
- Rank by leverage. Order the omitted variables by how much they move the decision, so scrutiny concentrates on the few that dominate.
The distinguishing move is that it works forward from ranges of inputs, not from a stored reference answer: it asks "if this suppressed thing were free to vary, would we still decide the same way," which no comparison against a single fixed baseline can tell you.
Tuning parameters¶
- Perturbation range — how wide a swing each variable is allowed. Too narrow and a real exposure hides inside the untested tail; too wide and everything looks fragile.
- One-at-a-time vs. joint — varying variables singly is cheap and readable; varying them together catches interactions but explodes the number of runs.[n1]
- Materiality band — how large a decision swing counts as "changed." Set it tight and trivial flips trigger alarms; set it loose and a genuine reversal gets waved through.
- Coverage vs. cost — how many suppressed variables get tested. Testing all is thorough but expensive; testing a screened subset is fast but can miss the one that mattered.
When it helps, and when it misleads¶
Its strength is that it converts a vague worry — "maybe we shouldn't have dropped that" — into a measured verdict with a magnitude attached, and it does so in priority order, so effort lands on the omissions that actually steer the outcome. It is the mechanism that earns the right to keep a simplification, by showing the dropped detail is genuinely inert across its range.
It misleads when the perturbation ranges are wrong or the interactions are ignored. A one-at-a-time sweep can pronounce every variable harmless while a combination of them flips the decision — the classic blind spot of local sensitivity analysis. It also silently assumes the simplified model's structure is right and only its inputs are in doubt; if the simplification dropped a whole mechanism rather than a variable, moving the surviving variables will never reveal it. The guarding discipline is to set plausible ranges from evidence rather than habit, to test the highest-risk variable combinations jointly, and to treat a "nothing moves" result as conditional on the tested ranges, not a universal all-clear.
How it implements the components¶
omission_relevance_criterion— it is the relevance test: an omission is judged relevant exactly when perturbing it swings the supported decision, and irrelevant when it does not.decision_consequence_threshold— the materiality band sets how large a swing counts as decision-changing, which is the threshold above which an omission must be treated rather than tolerated.residual_simplification_risk— for every variable left simplified after the sweep, it quantifies the residual exposure — how much the decision could move within the untested-but-bounded remainder.
It does not measure error against a fixed reference answer (comparison_baseline, preserved_function_or_invariant) — that forward-vs-exact comparison is Approximation Validation, its nearest twin; the difference is that Approximation Validation checks output error against a known-correct value, while this mechanism checks whether the decision flips when suppressed inputs are freed to vary.
Related¶
- Instantiates: Simplification Audit — it supplies the measured relevance verdict the audit uses to keep, caveat, or widen a simplification.
- Consumes: Omission Checklist — the inventory of suppressed variables it perturbs.
- Sibling mechanisms: Omission Checklist · Approximation Validation · Backtest Against Full Cases · Model Simplification Audit · Simplification Review · Stakeholder Review · Assumption Audit · Edge-Case Testing · Red-Team Review
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Sensitivity Check operates as an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution because it varies the variables a simplification fixed or dropped to see whether the decision it supports actually changes — separating omissions that are harmless from ones that are decision-critical.
Independent corroboration: The frozen evidence defines Sensitivity Check as 'Varies the variables a simplification fixed or dropped to see whether the decision it supports actually changes — separating omissions that are harmless from ones that are decision-critical', so its operative form is Analysis, Modeling & Optimization.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Operations Research
Origin pattern: Convergent development
Present-day reach: Universal
Rationale: A targeted perturbation check asks whether a modeled conclusion survives plausible input changes, a standard operations-research robustness operation. NASA and NIST formalize output response to input variation; statistics supplies inference and uncertainty treatment.
Related originating lineages:
- Data Science & Analytics — Data science, analytics, and operational monitoring supplies a parallel or contributing lineage for the mechanism's defining operation: varies the variables a simplification fixed or dropped to see whether the decision it supports actually changes — separating omissions that are harmless from ones that are….
- Engineering & Design — Safety factors and tolerance studies test whether simplified models preserve design decisions.
- Mathematics — Perturbation analysis supplies the formal question of how outputs change under altered assumptions.
- Statistics & Experimental Design — Robustness checks distinguish material modeling choices from harmless approximations.
Review resolution: The blind reviewers disagree on primary lineage (operations_research versus statistics_experimental_design). Authoritative or primary research supports operations_research as the best historical origin: A targeted perturbation check asks whether a modeled conclusion survives plausible input changes, a standard operations-research robustness operation. NASA and NIST formalize output response to input variation; statistics supplies inference and uncertainty treatment. The cited NIST, Guide for the Use of the International System of Units: Model Sensitivity; NASA, Sensitivity Analysis Overview directly supports the mechanism's defining operation. All independently supported contributing domains are retained without an arbitrary cap. origin_mode=convergent records the lineage relationship, while domain_reach=universal records later applicability separately from provenance.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
- NIST, Guide for the Use of the International System of Units: Model Sensitivity
- NASA, Sensitivity Analysis Overview
Notes¶
[n1] This is the standard distinction in sensitivity analysis between local, one-at-a-time methods and global methods (as systematized in the work of Andrea Saltelli and colleagues), the latter varying inputs jointly precisely because interaction effects are invisible to one-at-a-time sweeps. ↩