Leakage Sensitivity Grid¶
Sensitivity analysis — instantiates Shared-Source Variance Isolation
Sweeps a ladder of assumed contamination strengths the source cannot be measured at, and reports the level at which each conclusion breaks.
A Leakage Sensitivity Grid stress-tests a conclusion against a ladder of assumed contamination strengths it cannot measure directly: for each hypothesized level of shared-source leakage — none, low, medium, high — it recomputes what the adjusted result would be and reports at which level the conclusion flips. Its defining idea is bounding under ignorance: where the source can't be cleanly estimated, you do not guess a single correction — you assume a range of them and ask how much leakage it would take to overturn the claim. It never fits the source from data; it posits it and finds the breakpoint.
Example¶
A team reports that a new document classifier beats the incumbent on four held-out metrics — accuracy, F1, calibration, and a fairness gap. All four are computed from the same preprocessing-and-splitting pipeline, and there is a worry that some target information leaked through feature scaling that was fit on the full dataset. They cannot measure the leakage exactly, so they build a sensitivity grid: assume 0%, 2%, 5%, and 10% of the apparent gain is leakage-driven, and recompute the margin over the incumbent at each rung. The grid shows that the accuracy and F1 advantages survive even at 10% assumed leakage, but the calibration and fairness improvements evaporate above roughly 3%. The reported conclusion is downgraded accordingly — the classifier is robustly better on accuracy, only conditionally better on fairness. The output is a breakpoint map, not a point estimate.
How it works¶
Choose a leakage parameter — the share of covariance or effect attributable to the shared source — and define a plausible ladder of its values. At each rung, apply the corresponding adjustment and recompute the conclusion of interest. Then report the rung at which each claim changes status. The distinctive move is that it explores assumed contamination rather than estimating it, so its honesty comes from spanning a range and naming the tipping point — much as an E-value[n1] asks how strong an unmeasured source would have to be to explain a result away.
Tuning parameters¶
- Ladder range and spacing — how high the assumed leakage goes and how finely it is stepped. Wider ladders find distant breakpoints; finer spacing pinpoints the tipping level but adds runs.
- Leakage parameterization — share of variance versus multiplicative bias versus additive offset. The parameterization must match how the source plausibly acts, or the grid answers the wrong question.
- Claim set swept — a single headline claim versus every sub-claim. Sweeping all of them shows which conclusions are fragile and which are bulletproof.
- Plausibility anchoring — leaving the ladder abstract versus marking which rungs are realistic. Anchoring turns "it flips at 8%" into "and 8% is implausibly high here," which is what makes the grid decision-relevant.
When it helps, and when it misleads¶
Its strength is that it is the right tool exactly when the source is real but weakly observed and precise correction is impossible: it converts "we're not sure how much leaked" into "the accuracy claim holds unless leakage exceeds a level we consider unrealistic," protecting a decision without pretending to a number the data cannot give. Its failure mode is that the grid is only as meaningful as the plausibility of its rungs — an unanchored grid that sweeps physically absurd levels can manufacture false fragility or false robustness, and a mis-parameterized leakage term stress-tests the wrong thing. The classic misuse is presenting a wide, unanchored grid as if every rung were equally likely. The guarding discipline is to always pair the grid with a judgment about which leakage levels are plausible and report the breakpoint against that range, not in the abstract.
How it implements the components¶
common_variance_adjustment_rule— the grid's rungs are a family of adjustment rules: each assumed leakage level is a pre-specified bound on the shared component, applied in turn.residual_claim_boundary— the breakpoint is the boundary in motion: it marks, per claim, the contamination level at which support is lost.
It assumes contamination rather than measuring it, so it does not estimate the shared component from data or precision-weight it (precision_weight_update) — that's its nearest twin [Common Factor or Random-Effect Model], which fits the source instead of positing it. It inspects no residual correlations (independence_diagnostic_panel) — that's [Residual Correlation Diagnostic] — and plants no known-null tripwire (negative_control_dimension) — that's [Negative-Control Outcome Probe].
Related¶
- Instantiates: Shared-Source Variance Isolation — it supplies the robustness check that bounds a conclusion when the source cannot be pinned down.
- Consumes: Source Variance Audit Matrix supplies the flagged shared sources whose plausible strengths the grid sweeps.
- Sibling mechanisms: Multitrait-Multimethod Matrix · Common Factor or Random-Effect Model · Residual Correlation Diagnostic · Negative-Control Outcome Probe · Batch, Rater, or Instrument Counterbalancing Protocol · Variance Partitioning Report
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Leakage Sensitivity Grid operates as a computation, comparison, model, or analytic representation used to infer, estimate, or choose because it sweeps a ladder of assumed contamination strengths the source cannot be measured at, and reports the level at which each conclusion breaks
Independent corroboration: The frozen evidence defines Leakage Sensitivity Grid as 'Sweeps a ladder of assumed contamination strengths the source cannot be measured at, and reports the level at which each conclusion breaks', so its operative form is Analysis, Modeling & Optimization.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Sensitivity-analysis traditions developed grids over unmeasured bias or contamination strength to locate where conclusions reverse.
Related originating lineages:
- Data Science & Analytics — Model-validation practice materially shaped systematic presentation of robustness thresholds across outputs.
Review resolution: Both independent reviews place the primary lineage in statistics_experimental_design. The queued differences (encyclopedia_synthesis_disagreement) concern secondary metadata rather than primary provenance. The final retains data_science only where a reviewer supplied a formative-lineage rationale; downstream application by itself is not treated as origin. origin_mode=cross_disciplinary_synthesis records the relationship among origin traditions, while domain_reach=multi_domain records application breadth separately. encyclopedia_synthesis=true reflects whether either reviewer identified a corpus-specific synthesis, and confidence=high preserves the more cautious evidence assessment.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] The E-value (VanderWeele and Ding) reports how strong an unmeasured confounder would have to be to fully explain away an observed association. A leakage sensitivity grid applies the same logic to shared-source contamination: instead of estimating the leak, it reports the strength of leak that would be needed to overturn each conclusion. ↩