Sensitivity to Case-Set Analysis¶

Robustness method — instantiates Structured Comparative Case Design

Re-runs the comparison while dropping, swapping, or adding cases, to see whether the conclusion survives the particular set of cases that happened to be chosen.

Sensitivity to Case-Set Analysis holds the method fixed and varies the inputs: it re-runs the comparison many times over slightly different case sets — leave one case out, swap a borderline case, add a plausible omitted one — and watches whether the conclusion holds or flips. A finding that survives every reasonable perturbation is robust; one that depends on the presence or absence of a single case is fragile, however confident the headline sounded. What makes it THIS mechanism is that its object of study is the case set itself: it is the comparative analogue of an influential-observation check, and it reasons over a fixed pool rather than growing one.

Example¶

A study of eight cities concludes that those which kept flat transit fares recovered ridership faster after a service disruption. The claim looks solid across the eight. Sensitivity analysis puts it under load. Leave-one-out, city by city, mostly leaves the effect intact — until one city is dropped and the relationship vanishes entirely. Inspection shows that city ran an unrelated free-transit pilot during the window, single-handedly driving the apparent fare effect. Swapping a borderline city in and out wobbles the result further. The finding does not survive its own case set: it is downgraded from "flat fares drove recovery" to "suggestive, but hinges on one atypical case — needs more before anyone acts on it." The analysis changed nothing about the cities and everything about how much weight the conclusion can bear.

How it works¶

The method applies systematic perturbations to the case set — drop-one (jackknife), swap borderline cases, add a defensible omitted case — and records, for each, whether the conclusion's direction and strength survive. It surfaces the pivotal cases: the ones whose inclusion or exclusion moves the result the most. Its distinguishing feature is that it never re-collects data or re-specifies the model; it stresses only the composition of the comparison, exposing conclusions that were riding on a lucky roster rather than on a real pattern.

Tuning parameters¶

Perturbation type — drop-one, swap, or add. Drop-one finds cases the result leans on; swaps and adds test what a different roster would have shown.
Perturbation scope — every case, or only the borderline and contested ones. Broad scope is thorough; targeted scope concentrates on the cases most likely to matter.
Flip threshold — how large a change in the conclusion counts as "fragile." A strict threshold flags more findings as shaky; a loose one lets marginal results pass.
Reporting stance — whether the full perturbation profile is shown or only a summary. Showing the whole profile is what keeps the analysis honest.

When it helps, and when it misleads¶

Its strength is that it exposes the conclusions that hang on a single case or a fortunate selection — exactly the failure that small-N comparison is most prone to and least able to see.^[1] Its own limit is that it can only test robustness to the cases you can construct or imagine: a whole type of case missing from the universe will never show up in any drop-one, so a result can look robust and still be selection-bound. The classic misuse runs the method backwards — perturbing until a stable-looking subset appears and reporting that, which is the precise inverse of its purpose. The discipline that guards against it is to fix the flip threshold in advance and report the full perturbation profile, not the most flattering slice of it.

How it implements the components¶

inference_scope_and_boundary_conditions — its output is a robustness-qualified claim: how much the conclusion can be trusted given how sensitive it is to the case set.
negative_and_deviant_case_rule — it surfaces the pivotal or influential cases that single-handedly drive or break the result, flagging which cases the inference actually rests on.

It does not produce the cross-case matrix it perturbs — that is Cross-Case Evidence Matrix Tool; it does not add genuinely new cases to extend a finding — that is Replication Case Sampling Cycle; and it does not re-investigate what a specific anomaly *means — that is Deviant Case Follow-Up Protocol.*

Instantiates: Structured Comparative Case Design — the analysis is the design's guard against conclusions that depend on the roster of cases rather than on the world.
Consumes: Cross-Case Evidence Matrix Tool supplies the assembled case set whose composition is perturbed.
Sibling mechanisms: Replication Case Sampling Cycle · Deviant Case Follow-Up Protocol · Most-Similar Systems Design · Most-Different Systems Design · Matched Case Pairing Protocol · Measurement Equivalence Audit · Within-Case Process Tracing · Rival Explanation Elimination Table · Case Selection Bias Audit · Case Universe Sampling Frame · Comparative Case Review Panel · Comparative Historical Timeline · Configurational Comparison Truth Table · Counterfactual Contrast Memo · Cross-Case Evidence Matrix Tool

Notes¶

Its concern — the influential case that drives a result — is not the same as the deviant case that breaks a pattern, which is the business of Deviant Case Follow-Up Protocol. A case can be influential without being anomalous (it fits the pattern yet carries the whole result), anomalous without being influential (it breaks the pattern but changes no conclusion), or both. Sensitivity analysis is agnostic about why a case matters; it only measures how much the conclusion depends on it.

References¶

[1] The leave-one-out (jackknife) idea of the influential case — an observation whose removal materially changes the estimate or conclusion. In small-N comparison, where a single case is a large share of the evidence, checking for influential cases is a basic robustness safeguard. ↩