Skip to content

Proper Scoring Rule Comparison

Probabilistic-forecast comparison — instantiates Stochastic Process Modeling and Validation

Ranks competing probabilistic forecasts with a scoring rule that is optimized only by honest, accurate distributions, so the model that genuinely predicts best cannot be beaten by hedging or overconfidence.

When two or more stochastic models each emit a full predictive distribution, you need a single number to decide which forecasts better — one that cannot be gamed. Proper Scoring Rule Comparison supplies it. A scoring rule is proper when its expected value is optimized only by reporting the true distribution, so no forecaster can improve its score by shading probabilities, hedging toward the middle, or feigning confidence. Scoring every competing forecast on the same events with such a rule (log score, CRPS, Brier) rewards calibration and sharpness jointly and yields a decision-relevant ranking that says which model to keep and which to retire. Its distinguishing feature is that it is inherently comparative and incentive-compatible — where its siblings interrogate one model's internals, this one adjudicates between models on out-of-sample performance.

Example

A utility runs two rival day-ahead electricity-demand models — a classical statistical one and a machine-learned one — each emitting an hourly predictive distribution. Over a held-out quarter, every forecast is scored with the CRPS, which rewards the whole distribution and penalizes both bias and mis-stated confidence in the units of demand itself.

The result overturns the obvious read. The machine-learned model has the lower average point error, so a naïve mean-squared comparison would crown it. But its CRPS is worse: it is over-confident on volatile evenings, and the proper score punishes those narrow-but-wrong distributions hard. Because CRPS is aligned with the operational cost of being surprised, the comparison says keep the statistical model — a verdict the point-error metric actively hid.

How it works

  • Match the rule to the loss. Choose a proper score whose penalties mirror the decision's real cost of error — log score for harsh tail penalties, CRPS for a units-of-outcome measure, Brier for binary events.
  • Score on shared, held-out events. Evaluate every competing model's forecast on the same out-of-sample outcomes so the comparison is fair.
  • Aggregate and compare. Average the scores; a lower expected score is better under a proper rule.
  • Decide keep/retire. The winner earns retention; the loser is flagged for revision or retirement — with a significance check so a within-noise gap is not read as decisive.

Tuning parameters

  • Choice of proper rule — log score punishes tail misses severely (one confident wrong call dominates); CRPS is gentler and in outcome units; Brier suits binary events. The choice encodes what kind of error you most fear.
  • Threshold weighting — weight the score toward the outcomes that matter (e.g., high-demand hours) so the ranking reflects the decision, not the easy bulk.
  • Evaluation window — fixed hold-out vs rolling origin; longer windows stabilize the estimate, shorter ones track regime change.
  • Significance test — how large a score gap counts as real rather than sampling noise.
  • Aggregation — a single pooled mean vs per-regime scores that reveal where one model beats another.

When it helps, and when it misleads

Its strength is a single, game-proof, decision-aligned criterion that grades calibration and sharpness together and drives an explicit keep/retire choice — immune to the hedging and over-confidence that fool point-error metrics.

Its failures come from mis-use of the rule. Choosing an improper or mismatched score quietly rewards exactly the hedging propriety was meant to forbid, or optimizes a loss no one cares about. Reading a small score gap as decisive when it is within sampling noise manufactures a false winner. And a proper score is relative, not absolute: it ranks models against each other, but the winner of a bad field is still bad.[1] The discipline is to verify the rule is proper, match it to the actual loss, test the significance of score differences, and always pair the comparison with an absolute calibration check.

How it implements the components

Proper Scoring Rule Comparison fills the model-selection side of the archetype's machinery:

  • stochastic_model_revision_and_retirement_rule — the score is the objective criterion that decides which competing model is kept and which is revised or retired, closing the model lifecycle on evidence.
  • stochastic_modeling_question_and_use — the rule is chosen to mirror the decision's loss, tying the whole comparison to what the model is for rather than to a generic fit metric.

It builds none of the models it ranks, and it does not produce a single model's calibration histogram — that is Probability Integral Transform Check — or its generative check — that is Posterior or Simulation Predictive Check; it consumes competing models' forecasts.

References

[1] A scoring rule is proper if its expected score is optimized by reporting the true distribution, and strictly proper if that optimum is unique — so honest forecasting is the dominant strategy and no probability-shading can beat it. The log score and CRPS are strictly proper; ranking is comparative, so a proper score must be paired with an absolute check to confirm the winner is any good. ↩