Skip to content

Calibration Curve Review

Model diagnostic — instantiates Adaptive Threshold Recalibration

Checks whether a score's predicted probabilities still match observed frequencies before anyone moves the threshold that sits on it.

Calibration Curve Review is a diagnostic aimed not at the boundary but at the score underneath it. Its defining question is narrow and prior to everything else: when the model says 0.70, does the event actually happen about 70% of the time? If the score has drifted out of calibration, then the misfit a team is feeling is coming from the measurement, not the cutoff — and moving the threshold would be treating the wrong problem. This mechanism exists to catch that mistake before it happens. It says, in effect, fix the ruler before you argue about where to draw the line.

Example

A regional weather service issues probability-of-precipitation forecasts, and a downstream flood-alert threshold fires whenever the forecast exceeds a set probability. Before anyone touches that alert cutoff, forecasters run a calibration review: they bin every forecast from the last two seasons and plot predicted probability against the observed rain frequency — a reliability diagram. The days forecast at ≈70% rained only ≈55% of the time; the curve sags below the diagonal across the whole upper range. That is the finding: the score has become over-confident, likely after a model update. Re-cutting the flood threshold would paper over a systematically inflated forecast. The right move is to recalibrate the probabilities first, then revisit the boundary on an honest score.

How it works

The distinguishing method is the reliability diagram: group predictions into probability bins, and for each bin compare the mean predicted value with the observed outcome rate. Points on the diagonal mean the score is honest; systematic departure — a bowed curve, a consistent over- or under-confidence — is a drift signal that the score-to-outcome mapping has moved. It is deliberately upstream: it produces no new threshold and expresses no preference about error costs. Its entire output is a verdict on whether the number being compared to the threshold can still be trusted.

Tuning parameters

  • Binning granularity — how finely predictions are grouped; finer bins reveal local miscalibration but get noisy with sparse data.
  • Calibration window — how much recent history to include; shorter tracks fresh drift, longer stabilizes the estimate.
  • Calibration metric — reliability diagram, calibration error, or a Brier-score decomposition; which summary you trust to declare "miscalibrated."
  • Stratification — whether to check calibration within regimes (season, region, model version) rather than only in aggregate, since averages can hide offsetting errors.

When it helps, and when it misleads

Its strength is that it isolates a specific, common failure — the score, not the boundary, has moved — that every threshold review is otherwise liable to misattribute. Its classic misuse is running it backwards: re-cutting the threshold to compensate for a miscalibrated score, which bakes the measurement error permanently into the decision rule and hides it from anyone who later inspects the boundary. A near-cousin error is trusting aggregate calibration while offsetting subgroup errors cancel out. The discipline is to treat a miscalibration finding as a mandate to fix the score,[1] and only then hand a trustworthy score to the mechanisms that choose the cutoff.

How it implements the components

  • monitored_variable_or_score — its whole subject is the score itself: defining it, and auditing whether its values still mean what they claim.
  • drift_detection_signal — a departure from the calibration diagonal is the drift signal that conditions around the score have changed.
  • threshold_performance_metric — calibration error / reliability serve as the performance measure it reports upstream of any threshold change.

It does not weigh the cost of false alarms against misses (false_positive_false_negative_review) — Precision / Recall Tradeoff Review and Receiver Operating Characteristic Review do — nor does it set a revised boundary (recalibration_rule), which Risk Score Threshold Recalibration and Diagnostic Cutoff Revision own.

Notes

Calibration Curve Review is a gate, not a decision: it can veto a threshold change ("your score drifted — recalibrate that first") but never authorize one. Keeping it separate from the boundary-setting mechanisms is exactly what prevents a broken score from being silently absorbed into a moved cutoff.

References

[1] A reliability diagram (calibration plot) compares predicted probability against observed frequency; a well-calibrated score lies on the diagonal. It is a standard diagnostic in forecasting and probabilistic classification, and the correct response to poor calibration is to recalibrate the score, not to relocate the decision threshold. ↩