Scoring Rule¶
Evaluate a probabilistic forecast after its outcome by mapping the report–outcome pair to a numeric loss or reward, with propriety governing whether truthful distributions are optimal in expectation.
Core Idea¶
A scoring rule is a numerical evaluation rule for probabilistic forecasts. It takes an issued probability distribution P and the outcome y that actually occurs, then returns a real-valued loss or reward S(P,y). The rule makes forecasts with different uncertainty shapes commensurable after the event while retaining the distributional claim that was made before it. A forecaster who said “rain probability 0.7” and one who said “0.5” are not judged only by whether rain occurred; their stated degrees of uncertainty determine different scores for the same realization.
Scope of Application¶
Scoring rules apply wherever complete predictive distributions are issued and later confronted with realized outcomes.
- Binary event forecasting. Rain, recession, equipment failure, disease onset, and default probabilities can be evaluated with Brier or logarithmic scores while preserving confidence rather than reducing forecasts to yes/no calls.
- Multicategory prediction. A probability vector over mutually exclusive classes can be scored against the realized category. The rule evaluates allocation of probability across all categories, not only the winning label.
- Density forecasting. Continuous outcomes such as temperature, demand, wind speed, or asset returns require rules defined on densities or cumulative distributions.
Clarity¶
The first clarity rule is to declare orientation. Literature uses both scores-as-rewards and scores-as-losses. In a reward convention, larger is better and propriety means truthful reporting maximizes expected score. In a loss convention, smaller is better and truth minimizes expected score. Multiplication by -1 converts conventions without changing rankings, but silently combining them reverses conclusions. This draft uses loss orientation unless explicitly noted.
Manages Complexity¶
A predictive distribution is high-dimensional: it expresses location, spread, asymmetry, tails, multimodality, and dependence. A realized outcome supplies only one sample from that claim. A scoring rule compresses the report–realization relation into one number that can accumulate across forecast cases. Without such a common map, forecasters can selectively emphasize whichever aspect makes their forecast look favorable after the fact.
Abstract Reasoning¶
Report–outcome separation. Freeze the report before observing the realization. Any score computed from a revised distribution answers a different question and permits hindsight leakage.
Expected-score audit. For candidate truth Q, compute E_Q S(P,Y) as a function of possible reports P. Check whether P=Q is a minimum under loss orientation and whether it is unique. This is the definitive propriety test, not the rule's name or intuitive appeal.
Knowledge Transfer¶
Within forecast evaluation, the same package transfers from rain probabilities to disease risks, credit defaults, demand densities, categorical classifiers, and multivariate trajectories. The outcome space and scoring family change, but the audit questions persist: What distribution was reported? When was it frozen? Which outcome occurred? What rule and orientation were used? Is propriety established for the class? How were cases weighted? What uncertainty surrounds the aggregate?
Relationships to Other Abstractions¶
Current abstraction Scoring Rule Domain-specific
Parents (1) — more general patterns this builds on
-
Scoring Rule is a kind of Evaluation Prime
evaluation. A scoring rule applies a criterion-bearing map to a bounded forecast and realized evidence, producing an interpretable score.
Hierarchy path (1) — routes to 1 parentless root
- Scoring Rule → Evaluation → Comparison → Self Checking
Neighborhood in Abstraction Space¶
Scoring Rule sits in a sparse region of the domain-specific corpus (79th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.
Family — Expectation, Retrospection & Evaluation Bias (12 abstractions)
Nearest neighbors
- Probability Distribution — 0.84
- Credal Set — 0.83
- Learnable Function Class — 0.82
- Random Variable — 0.82
- Kaplan–Meier estimator — 0.82
Computed from structural-signature embeddings · 2026-09-08