Calibration Exercise¶

Predict-then-reveal calibration method — instantiates Competence Calibration Feedback

Has people commit a confidence estimate before the outcome is revealed, then repeats, so the running gap between stated confidence and actual result becomes visible and trainable.

Calibration Exercise trains the confidence-to-evidence mapping itself, by forcing the estimate to be committed before the answer is known and then repeating until a track record accumulates. Its defining move is pre-registration: you say "I'm 80% sure" and lock it, then the outcome arrives, and over many such items a pattern emerges — are your 80%s actually right about eight times in ten? Where a Reflective Error Log mines errors that already happened, this mechanism manufactures the prediction moment deliberately and in volume, so miscalibration becomes a measured curve rather than an anecdote. The unit of value is not any single prediction but the trend across many.

Example¶

A product-analytics team keeps confidently promising ship dates and delivery probabilities that don't hold up. They start a standing calibration exercise: for each upcoming feature, every analyst records a probability that it ships by the target date — a real number, committed before the sprint. Weeks later the outcomes land and each prediction is scored. Plotted across a quarter, the pattern is unmistakable: the events they called "90% likely" happened only about 60% of the time. Their point estimates weren't the problem; their confidence was systematically inflated. Seeing their own calibration curve does what no lecture about overconfidence could — the next quarter's "90%" starts meaning something closer to 90%. The exercise scores accuracy with a proper rule so that hedging everything to "50%" earns no credit,^[1] which keeps analysts honestly discriminating rather than safely vague.

How it works¶

Three features distinguish it from ordinary feedback. Pre-commitment: the confidence is fixed before the outcome, so it cannot be quietly revised in hindsight. Volume and repetition: many items, because a calibration curve needs a distribution, not a data point. Aggregation into a curve: predictions are bucketed by stated confidence and checked against realized frequency, turning "you're overconfident" into "your 90% band resolves at 60%." The exercise trains the mapping by exposing it, again and again, on scoreable items.

Tuning parameters¶

Confidence format — probabilities, numeric ranges, or ordinal bands. Probabilities enable proper scoring but demand numeracy; bands are gentler but coarser.
Item volume and spacing — how many predictions, how often. More items sharpen the curve; too many fatigue and the estimates go reflexive.
Scoring rule — Brier, log score, or hit-rate within stated ranges. Different rules punish overconfidence and vagueness differently; pick the one whose incentive matches the domain.
Feedback latency — how quickly outcomes return. Short loops train fast; long-horizon domains need proxy outcomes or patience.
Aggregation grain — per-person versus per-team calibration curves, and whether to break them out by topic.

When it helps, and when it misleads¶

Its strength is unique among the siblings: it is the only one that trains the confidence-to-evidence mapping through repetition, and it converts overconfidence from a felt accusation into a measurable, improvable curve.

It misleads when the items aren't representative — calibration drilled on trivia or toy forecasts may not transfer to the real domain that matters. A scoring rule can be gamed by hedging everything toward the middle, which looks calibrated but abandons discrimination. And a single sitting proves almost nothing; only the trend is informative. It is worth remembering, too, that calibration is not competence: a well-calibrated forecaster knows precisely what they don't know, which is valuable but different from knowing more. The discipline is to use domain-representative items, score across many trials, and keep calibration (are your confidences accurate?) distinct from resolution (do your predictions actually discriminate?).

How it implements the components¶

Calibration Exercise fills the prospective self-assessment-and-gap slice:

self_assessment — captured as an explicit, pre-outcome prediction, the exercise's raw input.
calibration_gap_map — the accumulated confidence-versus-outcome record is the gap map, in its cleanest quantitative form.
recalibration_cadence — repetition across many items is what turns a snapshot into an improving trend.

It does not deliver the explanatory feedback that interprets the curve (calibration_feedback, Benchmarked Feedback) or tie the result to what a person is allowed to do (decision_rights_boundary, Decision Rights by Competence).

Instantiates: Competence Calibration Feedback — produces the measured confidence-versus-outcome record the rest of the loop reads.
Sibling mechanisms: Reflective Error Log · Confidence Rating Scale · Benchmarked Feedback · Calibration Conversation · Exemplar Comparison · Competency Framework · Skills Assessment · Peer Review · Simulation or Case Test · Decision Rights by Competence · Supervised Practice

Notes¶

Calibration and competence are different axes. This exercise improves the accuracy of a person's confidence, not necessarily the quality of their underlying skill — someone can become well-calibrated about being mediocre. That is still valuable (it makes their self-reports trustworthy), but it is not a substitute for the skill-building that Supervised Practice or deliberate error work supplies.

References¶

[1] The Brier score is a proper scoring rule for probabilistic predictions: it rewards being both confident and correct and penalizes confident errors, so it cannot be gamed by hedging everything to 50%. "Calibration" in this tradition means stated probabilities match observed frequencies — your 70%s come true about 70% of the time. ↩