Calibrated Probability Elicitation¶
Test / assessment — instantiates Structured Expert Judgment Iteration
Elicits ranges, probabilities, or distributions while checking for overconfidence, incoherence, and calibration problems.
Calibrated Probability Elicitation treats an expert's numbers as measurements to be validated, not opinions to be recorded. Its defining move is to slip in questions whose answers are already known — seed questions — and score how well each expert's stated confidence matches their actual hit rate, so that an assessor's "I'm 90% sure" can be checked against how often they are right when they say 90%. Where a plain survey asks for a probability and writes it down, this assessment first asks whether the person giving it is even calibrated, then shapes the elicitation to squeeze out the overconfidence and internal incoherence that raw self-report is riddled with. The output is a tested distribution — a range or probability the assessor can trust because the person producing it demonstrably knows the difference between 60% and 95%.
Example¶
An intelligence unit is assessing the likelihood that a rival state fields a new capability within eighteen months. Rather than ask analysts for a gut percentage, the lead runs a calibrated elicitation. Each analyst first answers a battery of seed questions with checkable answers — "what fraction of the world's cargo tonnage transits this strait?", "in what year did this treaty enter force?" — supplying not just an answer but a 90% confidence interval. Scoring reveals the pattern the unit half-suspected: their most assertive analyst's "90%" intervals contain the truth only about 55% of the time. He is systematically overconfident, and now there is a number that says so.
Only after that calibration check does the frame turn to the real question, elicited the same disciplined way: not a single figure but a distribution, with the analyst forced to state at what point they would be surprised to be wrong. The overconfident analyst's intervals are widened to reflect his measured miscalibration before they enter the pool. The unit ends with a distribution over the eighteen-month question whose spread has been earned, not asserted.
How it works¶
- Seed with knowns. Pose questions with verifiable answers in the same format as the target question, and require a confidence range on each.
- Score calibration. Compare stated confidence to actual accuracy — a proper scoring rule such as the Brier score[1] rewards being right and being appropriately uncertain, penalizing both timidity and bravado.
- Elicit the target as a distribution. Ask for quantiles, intervals, or a full probability curve, using techniques (fixed points, "surprise" thresholds) that fight the instinct to give a single confident number.
- Coherence-check and adjust. Test that stated probabilities obey basic rules (they sum, they respect ordering) and, where the classical model is used, weight or recalibrate each expert by their seed performance.
Tuning parameters¶
- Seed battery size — more seed questions give a sharper calibration estimate but tax expert patience and can feel like a quiz rather than a judgment.
- Scoring rule — Brier, logarithmic, or interval scores differ in how harshly they punish overconfidence; a harsher rule pushes assessors toward humility, sometimes to a fault.
- Quantiles elicited — eliciting just a median and an 80% interval is fast; eliciting many quantiles yields a fuller distribution but multiplies the questions and the fatigue.
- Recalibration strength — how aggressively an expert's raw numbers are stretched or reweighted by their seed score. Strong correction fixes bias but can over-punish someone whose seed miss was noise.
When it helps, and when it misleads¶
Its strength is turning "how confident are you?" from rhetoric into a measured, comparable quantity, and catching the single most common defect in expert numbers: overconfidence that narrows intervals until they exclude the truth. A calibrated elicitation can also justify trusting a quiet, well-calibrated assessor over a loud, miscalibrated one on evidence rather than volume.
Its failure mode is assuming calibration transfers. Someone perfectly calibrated on almanac trivia may be wildly overconfident on their own specialty, where motivated reasoning bites hardest, so a good seed score can launder a bad substantive judgment.[1] The classic misuse is treating the resulting probabilities as measurement-grade precision — a 0.63 that implies data where there is only a disciplined guess. The guarding discipline is to choose seed questions close to the target domain, report the distribution's spread honestly, and keep the elicited number labeled as calibrated judgment, not fact.
How it implements the components¶
calibration_seed_question— the known-answer seed battery is the mechanism's signature; it is what makes this an assessment rather than a survey.uncertainty_distribution— the target is elicited as a range or probability curve, and calibration scoring directly tunes that spread.judgment_frame— it fixes the exact quantities, scales, and uncertainty format experts must answer in, so that "90%" means the same thing for everyone.
It scores individuals; it does not run the multi-pass loop or report where the panel converged — iteration_round, stopping_rule, and convergence_disagreement_report are Delphi Study's — nor does it visualize the pooled result over time (anonymized_feedback playback surfaces belong to Judgment Aggregation Dashboard).
Related¶
- Instantiates: Structured Expert Judgment Iteration — it supplies the calibrated, distribution-shaped inputs the rest of the process iterates on.
- Sibling mechanisms: Anonymous Survey Round · Delphi Study · Expert Elicitation Protocol · Judgment Aggregation Dashboard · Rationale Coding Matrix · Structured Forecasting Panel · Policy Expert Panel Process · Technical Consensus Round
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Elicits ranges, probabilities, or distributions while checking for overconfidence, incoherence, and calibration problems, making its operative form a bounded evaluation of existing evidence or work that produces a finding or disposition.
Independent corroboration: The frozen evidence defines Calibrated Probability Elicitation as 'Elicits ranges, probabilities, or distributions while checking for overconfidence, incoherence, and calibration problems', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Probabilistic forecasting and structured expert-judgment methods elicit distributions and test their coherence and calibration with seed questions and proper scores.
Related originating lineages:
- Operations Research — Decision analysis supplies formal expert elicitation for consequential choices under uncertainty.
- Psychology — Judgment research supplies the overconfidence and incoherence diagnostics that shape the questioning protocol.
Review resolution: Statistics is the agreed primary lineage because probabilistic forecasting and proper scoring formalized eliciting explicit probabilities. Operations research and psychology materially contribute decision use and judgment-bias controls; the mechanism remains an established statistical lineage.
Review outcome: Reconciled after independent review; high confidence.
References¶
[1] A proper scoring rule (the Brier score, introduced by Glenn Brier in 1950, is the canonical one) rewards a forecaster for stating probabilities that match observed frequencies, penalizing overconfidence and underconfidence alike. It measures calibration on the seed questions — but calibration is domain-sensitive, so a strong seed score is necessary, not sufficient, evidence that the target judgment is trustworthy. withdrawn registry ↩a ↩b