Finite-Sample or Exact Interval Check¶
Method — instantiates Coverage Probability Calibration
Replaces an asymptotic interval formula with an exact or small-sample-corrected construction that provably honors the nominal level at finite n, and compares the two side by side.
Finite-Sample or Exact Interval Check is the one method in this family that never resamples and never simulates — it reaches for a closed-form interval whose coverage is guaranteed by derivation rather than estimated by counting. Its defining move is analytic substitution: where an asymptotic formula only promises the nominal level as sample size goes to infinity, this check swaps in an exact construction (or a small-sample correction) whose containment is provable at the finite n you actually have, then holds the two intervals up against each other to expose how far the approximation drifts. Because coverage comes from a theorem, there is no Monte Carlo error and no dependence on a fitted model or a lucky sample. The price is scope: exact constructions exist only for particular estimands and distributions, and they are often conservative — they can over-cover to be safe.
Example¶
A contract manufacturer inspects a small batch and finds 2 defective units in a sample of 40. The reported figure is a 95% confidence interval for the defect rate, produced by the textbook Wald formula (estimate ± 1.96 standard errors). But with only two defects, the Wald interval is notoriously unreliable — it can even dip below zero — because the normal approximation it rests on has not kicked in at n = 40 with a rare event. The check replaces it with the Clopper–Pearson exact binomial interval, derived directly from the binomial distribution so that its coverage is guaranteed to be at least 95% for every true defect rate, and lays the two side by side.
The exact interval is wider and sits entirely in positive territory; the Wald interval was both too narrow and mislocated. Because the exact interval's coverage is a proven property of the binomial, not an estimate, the manufacturer can quote it without a simulation study — and can see, from the gap between the two, exactly how much the asymptotic shortcut was understating uncertainty for a small, low-defect batch.
How it works¶
- Identify the asymptotic assumption at risk. Locate where the reported interval leans on large-n normality — small samples, rare events, boundary parameters.
- Find the exact or corrected construction. Substitute an interval derived to hold at finite n — an exact distributional interval (e.g., Clopper–Pearson), a small-sample correction, or a continuity/degrees-of-freedom adjustment.
- Compare intervals, not just endpoints. Place the asymptotic and exact intervals side by side; the discrepancy quantifies the approximation error the formula was hiding.
- Prefer the provable guarantee. When the exact interval is available and the stakes justify its width, report it — its coverage is a theorem, not a measurement.
Tuning parameters¶
- Exactness vs. width — exact intervals guarantee the nominal level but are frequently conservative; a mid-p or score interval trades a little of the guarantee for tighter width.
- Which correction — Clopper–Pearson vs. Wilson score vs. Agresti–Coull for a proportion, or a t vs. z pivot elsewhere; each balances guaranteed coverage against length differently.
- Guarantee target — whether you require coverage ≥ nominal for every parameter value (strict) or on average across a range (relaxed); strictness costs width.
- Applicability gate — how far you will stretch a known exact result to a non-standard estimand before the derivation no longer holds and the guarantee evaporates.
When it helps, and when it misleads¶
Its strength is certainty without computation: when an exact construction exists, coverage is settled by proof, immune to the Monte Carlo error of a simulation and the model risk of a parametric audit. In small-sample, rare-event, and boundary situations — exactly where asymptotics fail — it is the most defensible interval available.
Its failure mode is over-conservatism and narrow applicability. Exact intervals can over-cover, buying a guarantee with excess width that frustrates decision-makers, and a strictly conservative interval is a real cost when precision matters.[1] The classic misuse is transplanting an exact result to an estimand it was never derived for — where the guarantee silently no longer applies — and then trusting it as if it did. The guarding discipline is to confirm the exact construction's assumptions genuinely match the problem, and to state plainly when the guaranteed interval is conservative rather than sharp.
How it implements the components¶
- finite_sample_correction — its whole substance is supplying the exact or small-sample-corrected interval that replaces a formula which only holds asymptotically.
- nominal_coverage_contract — an exact construction honors the nominal level as a proven property at finite n, restoring the contract that "95%" means at least 95% containment rather than "95% eventually, for large enough samples."
It neither simulates nor resamples: the known-truth simulation coverage_performance_test is Monte Carlo Coverage Simulation and the assumption-light resampling_benchmark is Nonparametric Resampling Interval Check, its nearest counting-based twin. It also does not stratify results by subgroup_coverage_slice — that is Subgroup Coverage Calibration Table.
Related¶
- Instantiates: Coverage Probability Calibration — it is the analytic route to a coverage guarantee when simulation is unnecessary or asymptotics are untrustworthy.
- Sibling mechanisms: Monte Carlo Coverage Simulation · Parametric Bootstrap Coverage Audit · Nonparametric Resampling Interval Check · Pre-Registered Simulation Grid · Subgroup Coverage Calibration Table · Calibration-Set Interval Adjustment · Historical or Holdout Coverage Backtest
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: The method constructs an exact or small-sample-corrected interval and analytically compares its finite-n coverage with the asymptotic alternative.
Nearest alternative: Assessment, Review & Assurance — Coverage is checked, but the operative form is the formal statistical construction and comparison rather than an audit process.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Exact and small-sample confidence intervals are canonical statistical-inference constructions developed to ensure finite-sample coverage.
Related originating lineages:
- Mathematics — Probability distributions and analytic derivations establish the guarantees.
Review resolution: Both reviewers agree that statistics_experimental_design is primary. I retain mathematics only as formative origin lineage(s), without treating every later application as an origin. single_lineage is appropriate because the evidence supports one principal professional lineage. Reach is multi_domain as a separate applicability judgment: it does not widen or narrow the recorded provenance. Encyclopedia synthesis is false because the artifact is already established enough that encyclopedia-specific synthesis is not required. The secondary differences are reconciled with no unresolved primary-provenance ambiguity.
Review outcome: Reconciled after independent review; high confidence.
References¶
[1] The Clopper–Pearson interval (1934) inverts the exact binomial test to guarantee coverage ≥ nominal for any true proportion; the Wilson score interval is a less conservative alternative. Both illustrate the exact-method trade-off — a provable guarantee bought, often, with more width than an approximate interval would use. registry ↩