Nearby-World Sensitivity Review¶
Validation procedure — instantiates Minimal-Change Counterfactual Construction
Perturbs the counterfactual into nearby coherent variants to check the conclusion survives small, defensible changes to what was held fixed.
The Nearby-World Sensitivity Review takes a chosen counterfactual and asks whether its conclusion is robust or a knife-edge. It generates a spread of nearby, coherent variants — small, equally-defensible perturbations of the edited and held-fixed facts — and re-evaluates the conclusion on each. Its defining move is sweeping a neighborhood and testing answer-stability across it: a conclusion that holds only for one precise construction, and flips under a trivial nudge, is fragile and should not be trusted. It uses the closeness metric to decide which perturbations count as "nearby" and to bound the neighborhood it sweeps. It does not tabulate the per-fact anatomy of any single comparison, and it does not build the master ranking of worlds — it reads that closeness order to define a neighborhood, then reports whether the answer survives inside it.
Example¶
A bank's model denies a loan application, and an automated system generates a counterfactual explanation: "you would have been approved if your annual income were about six thousand dollars higher." Before that single explanation is handed to the applicant as actionable advice, the Nearby-World Sensitivity Review perturbs it into nearby coherent variants. Income raised by a slightly smaller amount, with debt-to-income shifting correspondingly; the same income change but starting from a marginally different recent-inquiries count; the raise combined with a small change in the reported credit line. Each variant stays inside a small closeness radius and remains internally coherent. The review re-runs the model on all of them. If tiny, equally-plausible variations flip the outcome back to "denied," the explanation sits on a knife-edge and is not trustworthy guidance. If "approved" holds across the whole nearby neighborhood, the counterfactual is robust. The review's output is a verdict on stability, plus the nearest variant that disagrees — not a new counterfactual.
How it works¶
- Take the selected counterfactual. Start from the chosen world whose conclusion is to be stress-tested.
- Generate nearby variants. Perturb edited and held-fixed facts by small amounts within a stated closeness radius, spanning the dimensions that could matter.
- Keep only coherent variants. Discard perturbations that produce an impossible or infeasible world.
- Re-evaluate on each. Re-run the conclusion across the whole set of nearby worlds.
- Report agreement and the failure boundary. State the fraction that agree and surface the nearest variant that flips the answer.
Tuning parameters¶
- Neighborhood radius — how far "nearby" extends. Too tight and everything looks robust; too wide and nothing does. The radius is the highest-leverage dial and the one most worth stating up front.
- Perturbation dimensions — which facts get jittered. Omitting the sensitive variable manufactures false confidence; including it is the whole point.
- Coherence filter — how strictly incoherent variants are discarded before evaluation, so robustness is measured only over genuinely possible worlds.
- Agreement threshold — what fraction of the neighborhood must agree for the conclusion to pass as robust.
- Failure-boundary reporting — whether the nearest disagreeing world is surfaced. Reporting it turns a pass/fail into a map of exactly how fragile the claim is.
When it helps, and when it misleads¶
Its strength is that it separates a robust counterfactual claim from a fragile one — the central worry behind counterfactual explanations in automated decisions, which are only actionable if they are stable to small, plausible changes in the inputs.[1] A conclusion that survives its whole neighborhood earns trust the single-point version cannot.
Its failure mode is a mis-set radius: a neighborhood drawn too small confirms robustness that is not there, while sweeping only along stable dimensions and skipping the sensitive one produces false reassurance. The signature misuse is declaring robustness after perturbing only the variables that happen not to matter. The guarding discipline is to state the radius and the perturbation dimensions before running the sweep, deliberately include the variables most likely to be sensitive, and always report the nearest failure so a passing verdict carries its own caveat.
How it implements the components¶
nearby_alternative_sensitivity_set— it generates, filters, and evaluates the spread of nearby coherent variants that constitutes the sensitivity set, then reports the conclusion's stability across it.closeness_ordering_rule— it uses the stated closeness metric to decide which perturbations qualify as "nearby" and to bound the neighborhood it sweeps.
It reads a closeness order rather than producing one: the master ranking of candidate worlds is built by Closest-World Ranking Table. And it sweeps a neighborhood of many worlds rather than dissecting one — the modification_trace_record that tabulates a single comparison cell by cell belongs to its nearest relative, Minimal-Difference Matrix, which anatomizes one difference where this review tests a whole spread of them.
Related¶
- Instantiates: Minimal-Change Counterfactual Construction — the Review certifies that a minimal-change counterfactual's conclusion is stable, not a coincidence of one exact construction.
- Consumes: Closest-World Ranking Table supplies the closeness order and top world the review perturbs around.
- Sibling mechanisms: Minimal-Difference Matrix · Closest-World Ranking Table · Entailment Cascade Audit · Ideal Baseline Rejection Note · Antecedent Edit Card
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Nearby-World Sensitivity Review operates as an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation because it perturbs the counterfactual into nearby coherent variants to check the conclusion survives small, defensible changes to what was held fixed.
Independent corroboration: The frozen evidence defines Nearby-World Sensitivity Review as 'Perturbs the counterfactual into nearby coherent variants to check the conclusion survives small, defensible changes to what was held fixed', so its operative form is Experiment, Test & Rehearsal.
Nearest alternative: Assessment, Review & Assurance — Nearby-World Sensitivity Review includes features of a bounded evaluation of existing evidence or work that produces a finding or disposition, but its defining operation is an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Philosophy
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Counterfactual philosophy supplied the nearby-possible-world idea and the demand that a counterfactual hold fixed all but defensible small changes.
Related originating lineages:
- Computer Science & Software Engineering — Machine learning operationalized nearby counterfactuals as actionable explanations and made local stability a practical validation concern.
- Economics & Finance — Economic counterfactual modeling materially developed comparative-statics checks over nearby assumptions.
- Statistics & Experimental Design — Sensitivity analysis supplies the disciplined perturbation and robustness check.
Review resolution: Both independent reviews agree on primary origin philosophy; reconciliation resolves reported_ambiguity, alternate_origin_disagreement. Formative alternate lineages retained: computer_science, economics_finance, statistics_experimental_design. The broader reach of later applications is kept separate as domain_reach=multi_domain; origin_mode=cross_disciplinary_synthesis describes the historical relationship among lineages. Confidence is conservatively reconciled to medium, and encyclopedia_synthesis=true preserves the reviewers' boundary judgment.
Attribution caveat: Possible-world proximity is philosophical, while the operational review resembles statistical sensitivity analysis.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; medium confidence.
References¶
[1] In machine learning, counterfactual explanations describe the smallest change to an input that would flip a model's decision (Wachter, Mittelstadt & Russell, 2017). Their usefulness depends on stability: an explanation that reverses under a tiny, plausible perturbation is a knife-edge, not actionable advice — which is exactly what this review tests for. withdrawn registry ↩