Subgroup Coverage Calibration Table¶
Artifact — instantiates Coverage Probability Calibration
A table that reports nominal versus realized coverage broken out by subgroup, site, period, or risk stratum, so local undercoverage cannot hide inside a healthy overall average.
A Subgroup Coverage Calibration Table is the artifact that refuses to let one average speak for everyone. It takes an interval procedure's realized coverage and disaggregates it — one row per subgroup, site, time period, tail state, or risk stratum — placing the nominal level beside the empirical coverage for each slice and flagging the rows that miss. Its defining idea is that marginal coverage can be perfectly calibrated overall while a consequential subgroup is badly undercovered; a 95% interval that covers 95% on average but 85% for a rare, safety-critical class is not calibrated for that class. The table is a reporting artifact built after the coverage is measured — its job is to expose where the promise holds and where it breaks, not to measure coverage from scratch or to repair it.
Example¶
A hospital deploys a model that outputs a prediction interval for a patient's 30-day readmission risk. Overall, the intervals look well calibrated — realized coverage sits near the nominal 90%. But a clinician worries the model was trained mostly on common presentations. The data team builds a coverage table stratified by age band, primary diagnosis category, and payer type, computing realized coverage within each cell against the same 90% target.
Most rows sit near 90% — but for patients under 40 with a rare diagnosis, realized coverage is about 74%, and the row is flagged red. That pocket of undercoverage was completely invisible in the global number, and it lands on exactly the group where an over-confident interval could mislead a discharge decision. The table hands that finding to the calibration loop as a specific, located problem — this subgroup, this gap — rather than a vague sense that "the model is fine on average."
How it works¶
- Choose the slicing dimensions. Pick the subgroups, sites, periods, or strata where coverage could plausibly differ and where a local failure would matter.
- Compute coverage per slice. Within each cell, count how often the interval contained the target and place that beside the nominal level.
- Flag the gaps. Mark rows whose realized coverage falls outside tolerance, and note the cell's sample size so a noisy small-n row is not mistaken for a real failure.
- Surface, don't average. Present the disaggregated rows rather than collapsing them, so a compensating over-coverage elsewhere cannot mask a local miss.
Tuning parameters¶
- Slice granularity — coarse strata give stable estimates but can still hide a failure inside a broad bucket; fine strata localize the problem but starve each cell of data and inflate noise.
- Intersectional depth — single-axis slices vs. crossed subgroups (age × diagnosis); crossing finds interaction failures but explodes the number of thinly-populated cells.
- Tolerance band — how far a slice may drift from nominal before it is flagged; a tight band catches more but raises false alarms on small cells.
- Small-cell handling — whether to suppress, pool, or widen the uncertainty on rows with too few cases; mishandling this turns sampling noise into phantom subgroup failures.
When it helps, and when it misleads¶
Its strength is defeating average-coverage masking — the failure mode where a global coverage number certifies a procedure that is quietly broken for a minority regime. Whenever fairness, safety, or regulatory validity depends on reliable intervals for particular groups, the disaggregated view is the only honest one, and it turns "coverage is fine" into a located, actionable claim.[n1]
Its failure mode is the small-cell trap: slice finely enough and every rare subgroup shows a wild coverage estimate driven by a handful of cases, so the table manufactures alarming rows that are pure noise. The classic misuse is reading a 60%-coverage cell built on eight observations as a real failure — or, conversely, choosing coarse strata that keep a genuine failure buried. The guarding discipline is to report each cell's sample size and uncertainty, and to distinguish a statistically real subgroup gap from sampling noise before acting on it.
How it implements the components¶
- subgroup_coverage_slice — the table is the per-slice coverage breakdown: one row of realized-versus-nominal coverage for each subgroup, site, period, or stratum.
- empirical_coverage_gap_diagnostic — each row computes the nominal-minus-realized gap and flags it, applying the coverage-gap diagnostic locally rather than only in aggregate.
It does not decide the slicing regimes ahead of time — that pre-commitment is Pre-Registered Simulation Grid, its sibling-artifact twin, which owns relevant_regime_set and coverage_reporting_note; this table reports realized gaps after the fact rather than pre-specifying conditions. It also neither runs the underlying test (coverage_performance_test, Monte Carlo Coverage Simulation) nor corrects the flagged rows (calibration_adjustment_rule, Calibration-Set Interval Adjustment).
Related¶
- Instantiates: Coverage Probability Calibration — it is the artifact that makes local undercoverage visible so the loop can act on it.
- Consumes: the per-case containment results from Monte Carlo Coverage Simulation or Historical or Holdout Coverage Backtest, which it stratifies.
- Sibling mechanisms: Pre-Registered Simulation Grid · Monte Carlo Coverage Simulation · Parametric Bootstrap Coverage Audit · Nonparametric Resampling Interval Check · Finite-Sample or Exact Interval Check · Calibration-Set Interval Adjustment · Historical or Holdout Coverage Backtest
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Subgroup Coverage Calibration Table is defined in the frozen evidence as: A table that reports nominal versus realized coverage broken out by subgroup, site, period, or risk stratum, so local undercoverage cannot hide inside a healthy overall average. Its operative deployed or enacted form is therefore Analysis, Modeling & Optimization.
Nearest alternative: Representation, Specification & Plan — Representation, Specification & Plan can support this mechanism, but the evidence centers the concrete operation described above rather than the alternative family's defining operation.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Nominal-versus-realized coverage by stratum is sampling calibration.
Related originating lineages:
- Data Science & Analytics — Tables expose gaps.
- Mathematics — Mathematical modeling, proof, and abstract-structure practice supplies a parallel or contributing lineage for the mechanism's defining operation: a table that reports nominal versus realized coverage broken out by subgroup, site, period, or risk stratum, so local undercoverage cannot hide inside a healthy overall average.
- Public Administration & Policy — Local undercoverage is inequitable.
- Ethics of Technology & AI Governance — Technology ethics and ai governance supplies a parallel or contributing lineage for the mechanism's defining operation: a table that reports nominal versus realized coverage broken out by subgroup, site, period, or risk stratum, so local undercoverage cannot hide inside a healthy overall average.
Review resolution: The blind reviewers agree that statistics_experimental_design is the primary origin and differ only on alternate origin disagreement, origin mode disagreement, domain reach disagreement, encyclopedia synthesis disagreement. I preserve every independently explained alternate from both records rather than imposing a numeric cap. I retain cross_disciplinary_synthesis because the combined evidence shows material contributions from several lineages. The broader reach of multi_domain records portability separately from historical provenance; encyclopedia_synthesis=true preserves the affirmative synthesis judgment where either reviewer identified one.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] The distinction between marginal coverage (the interval covers at the nominal rate averaged over the whole population) and conditional coverage (it covers within each subgroup) is central to modern interval calibration; conditional or "equalized" coverage guarantees are strictly stronger and are what subgroup tables audit for. ↩