Skip to content

Confidence Interval

Statistical method — instantiates Uncertainty Explicitness

Replaces a single exact-looking estimate with a range produced by a stated procedure, so the sampling uncertainty around the number travels with the number instead of being rounded away.

A Confidence Interval is the archetype's quantitative representation for uncertainty that arises from sampling: instead of reporting an estimate as one exact number, it reports a range computed by a defined procedure, at a stated confidence level (commonly 95%). Its distinguishing move is that the interval is a property of the estimation procedure, not a hand-drawn hedge — its width is derived from the data's variability and the sample size, so it cannot be widened or narrowed to taste. That derivation is exactly what makes it more disciplined than a qualitative label and more honest than a bare point estimate: the number's precision is pinned to the evidence that produced it.

Example

A team runs an A/B test on a checkout redesign. The raw result looks decisive: the new variant converts at 4.2% versus 3.9% — "a 0.3-point lift." Reported that way it reads as fact. Computing a 95% confidence interval on the difference instead yields something like "+0.3 points, 95% CI roughly −0.1 to +0.7." The same experiment now tells a truthful story: the best estimate is a lift, but the interval straddles zero, so the data cannot yet rule out no effect at all. The width comes straight from the sample size — run the test longer and the interval narrows. The team holds the launch and keeps collecting, a decision the point estimate alone would have hidden behind a confident-looking 0.3.

How it works

From a sample, the method estimates both the quantity of interest and its variability, then combines them — through the sampling distribution — into a range at a chosen confidence level. What distinguishes it from its siblings: the width is computed, not asserted; it shrinks predictably as the sample grows; and it aggregates the identifiable contributors to sampling error into one stated interval, an uncertainty budget for the estimate. The confidence level is the long-run coverage of the procedure — the fraction of intervals that would contain the true value across many repetitions — which is a precise claim, and a frequently mis-stated one.[1]

Tuning parameters

  • Confidence level — 90%, 95%, 99%. Higher confidence widens the interval; it buys a stronger coverage guarantee at the cost of a less precise-looking range.
  • Method — normal approximation, exact, or bootstrap. The approximation is simple but degrades for small samples or skewed data, where a bootstrap or exact interval is more faithful.
  • One- vs two-sided — whether you care about a bound in one direction or a range around the estimate.
  • Reported precision — how many digits the endpoints carry; reporting an interval to spurious decimals reintroduces the false precision the interval was meant to cure.
  • What variability is included — sampling error only, or an attempt to fold in other error sources; a narrow interval that ignores known biases understates the true uncertainty.

When it helps, and when it misleads

Its strength is discipline: the range is a consequence of the data, not a matter of opinion, so a wide interval is an honest signal that the evidence is thin. Its failure modes are mostly about interpretation. The confidence level is routinely misread as the probability the true value lies in this one interval, which it is not.[1] An interval captures only sampling uncertainty and says nothing about a biased sample, a mismeasured instrument, or a wrong model — so a tight interval on bad data looks reassuringly precise while being confidently wrong. And it is easily run backwards: a level chosen, or endpoints rounded, to make a result cross a significance line already desired. The discipline that guards against this is to state the procedure and confidence level plainly, to remember the interval bounds only sampling error, and to resist reading a narrow interval as evidence of a correct answer.

How it implements the components

  • confidence_representation — the interval is the confidence representation: a numeric statement of how much the estimate could vary, at a stated level.
  • precision_discipline — by reporting a range whose width is fixed by the data, it stops a single point estimate from claiming more exactness than the sample supports.
  • uncertainty_budget — its width aggregates the identified contributors to sampling error into one accounted-for quantity, an explicit budget for the estimate's uncertainty.

It does not translate this into a plain-language tag or a visual mark (stakeholder_uncertainty_translation, visual_uncertainty_encoding) — that is Confidence Label and Error Bar — and it does not check whether its stated coverage actually holds over time, which is Probability Estimate's calibration loop.

  • Instantiates: Uncertainty Explicitness — it makes sampling uncertainty explicit and pins an estimate's precision to its evidence.
  • Sibling mechanisms: Error Bar · Probability Estimate · Confidence Label · Uncertainty Band · Forecast Range · Risk Register · Assumption Register · Caveated Decision Memo · Evidence Grade Rubric · Known Unknowns Log · Model Limitations Card

Notes

An Error Bar is often just a confidence interval drawn: the method here produces the number, the error bar renders it on a chart. Keeping the two straight matters because a figure's error bars are frequently not confidence intervals but standard deviations or standard errors — the same visual mark, a very different meaning.

References

[1] A 95% confidence interval does not mean there is a 95% probability the true value lies within this particular interval. The 95% refers to the procedure's long-run coverage — across many repeated samples, about 95% of the intervals it produces would contain the true value. Any given interval either contains it or does not; the probability statement is about the method, not the single result. ↩