Skip to content

Probability Integral Transform Check

Calibration diagnostic — instantiates Stochastic Process Modeling and Validation

Feeds each observation through its own predicted cumulative distribution; if the forecasts are calibrated the transformed values are uniform, so departures from flatness reveal exactly how the distribution is wrong.

There is a small, exact fact that makes distributional calibration testable: if a random outcome truly follows a distribution with cumulative function F, then F evaluated at the outcome is Uniform on [0,1]. Probability Integral Transform Check turns that into a diagnostic — feed every observation through its own predicted CDF, collect the transformed values, and check the pile for uniformity. A flat histogram means the predictive distributions are calibrated; a non-flat one is not merely a "fail" but a shaped verdict that says how the model is wrong. Its distinguishing power is diagnosis by shape: it reads the error, rather than only detecting it, and it grades the whole predictive distribution, not just its mean.

Example

A weather service issues a full probabilistic forecast of next-day rainfall each morning — a distribution, not a single number. Over a season, take each day's observed rainfall and read off where it fell in that day's forecast CDF, giving one value per day. Pile those into a histogram.

The shape is the diagnosis. A flat histogram would mean the forecasts are honestly calibrated. Instead it comes out U-shaped: too many observations land in the extreme buckets, meaning reality keeps landing outside the forecast's confident middle. The forecasts are under-dispersed — systematically over-confident, assigning too little probability to surprises. A hump in the middle would say the opposite (too timid); a lopsided tilt would flag a wet or dry bias. The ensemble-forecasting version of exactly this plot is the rank histogram.

How it works

  • Transform each observation. Compute the predicted-CDF value at each realized outcome — its PIT value. Under a correct predictive distribution these are independent Uniform(0,1).
  • Test for uniformity. Inspect the histogram or empirical CDF of the PIT values for flatness, by eye or with a uniformity test.
  • Read the deviation shape. U-shape ⇒ forecasts too narrow (over-confident); central hump ⇒ too wide (under-confident); slope ⇒ directional bias.
  • Handle discrete or ensemble forecasts with a randomized PIT or the equivalent rank histogram, so ties don't spoil the uniformity.

Tuning parameters

  • Histogram resolution — bin count trades sensitivity to fine miscalibration against noise in each bar.
  • Randomized vs standard PIT — for discrete outcomes, randomization restores uniformity that ties would otherwise break.
  • Uniformity test — a formal test (Kolmogorov–Smirnov, chi-square) versus a graphical read; the plot shows shape, the test gives a number.
  • Conditioning — compute PIT overall, or split by regime/covariate to locate where the model is miscalibrated rather than averaging it away.
  • Independence check — whether to test the PIT values for serial dependence, not just uniformity.

When it helps, and when it misleads

Its strength is a sharp, diagnostic calibration test: it says not merely that the predictive distribution is wrong but how — too confident, too timid, or biased — grading the full distribution rather than a point forecast.

Its central trap is that a flat histogram is necessary but not sufficient. PIT values must also be independent; a model can be marginally calibrated yet mis-time its uncertainty, passing the flatness check while failing in sequence. Worse, averaging over regimes can hide compensating miscalibrations — over-confident in calm periods and under-confident in storms can cancel to a deceptively flat overall histogram.[1] The discipline is to test the PIT values for independence as well as uniformity, and to condition on regime so that opposite errors cannot mask each other.

How it implements the components

Probability Integral Transform Check fills the distributional-calibration side of the archetype's machinery:

  • stochastic_calibration_plan — it produces the calibration verdict: whether the model's stated probabilities match observed frequencies, and in which direction they miss.
  • probability_law_specification — it operates directly on the specified predictive law, testing its whole shape — tails included, not just mean and variance — via the uniformity of the transform.

It does not simulate replicate paths and compare summary features — that is Posterior or Simulation Predictive Check — rank competing models — that is Proper Scoring Rule Comparison — or test the process's residual independence, which is Residual Independence and Whiteness Test; it consumes a model's predictive distributions.

References

[1] The probability integral transform is the exact fact that F(X) is uniform when F is the true distribution of X; its ensemble-forecast counterpart is the rank histogram. A calibrated forecast is a necessary virtue but must be paired with sharpness — the narrowest distribution that stays calibrated — since a vague forecast can be perfectly calibrated yet useless. ↩