Expert Disagreement Calibration¶
Comparison method — instantiates Heuristic Calibration and Confidence Judgment
Uses the spread among several independent experts on the same case as a live reliability signal — high disagreement caps confidence and triggers escalation.
Expert Disagreement Calibration turns the spread between independent judges into a calibration instrument. When you cannot yet observe the outcome — the diagnosis is fresh, the forecast unresolved — you still have a proxy for how trustworthy a heuristic judgment is on this case: whether several qualified people, judging independently, land in the same place. Tight agreement suggests the cues are clear and the heuristic is on firm ground; wide disagreement suggests the case is ambiguous or outside the rule's competence, and confidence should be capped no matter how sure any single judge feels. Its defining move is that the signal is cross-sectional and outcome-free — it reads reliability off the present divergence among peers rather than off a history of resolved calls. Disagreement is not a problem to be voted away; it is the measurement.
Example¶
A hospital routes chest-radiograph reads through a fast-track heuristic where one radiologist's confident call clears the image. To calibrate, the department periodically has three radiologists read the same batch independently, blind to each other. On most images all three agree and are confident — the fast track is well-earned there. But a subset draws a three-way split: one calls it clearly normal with high confidence, another flags a possible early nodule, the third is unsure. That divergence, not any single read, is the finding: on cases with this cue pattern, individual confidence is unreliable, because equally qualified peers cannot converge. The department writes the rule accordingly — images matching that ambiguous pattern have their single-reader confidence capped and are automatically sent for a second independent read before the fast track applies. Where all three converge, the single confident read stands. The disagreement rate becomes a standing gauge of where the heuristic may and may not carry solo confidence.
How it works¶
Several judges evaluate the same cases independently and blind — independence is the whole game, because once judges see each other's calls, social influence collapses the spread and destroys the signal. Their judgments (and, ideally, each judge's own confidence) are collected and the divergence measured: how far apart the calls are, and whether disagreement clusters on particular case types. High-spread cases are flagged as low-reliability zones. The method then binds that finding to action in two ways: it caps the confidence that any single judge's call may carry on high-disagreement case types, and it escalates individual cases whose reads diverge to a second opinion, panel, or adjudication. Structured elicitation — asking each expert to commit before comparing, sometimes across rounds — keeps the divergence honest rather than herded.[n1] The output is a map of where peers converge (solo confidence allowed) and where they scatter (confidence capped, escalation required).
Tuning parameters¶
- Panel size — how many independent judges per case. More judges estimate the spread more reliably but multiply the expensive expert time the heuristic was meant to save.
- Disagreement threshold — how much divergence caps confidence or forces escalation. A tight threshold catches ambiguity early but floods the second-opinion queue; a loose one preserves throughput but lets shaky solo calls through.
- Independence enforcement — how strictly judges are kept blind to one another. Strict blinding protects the signal; relaxed blinding is cheaper but risks herding that fakes agreement.
- Confidence weighting — whether each judge's own stated confidence factors into the spread, or only their point call. Including it is richer but rewards judges who are merely more assertive.
When it helps, and when it misleads¶
Its strength is that it works before outcomes exist — for slow-to-resolve or never-resolved judgments where a track record can't be built, peer divergence is often the only reliability signal available, and it is a good one. It also catches case-specific ambiguity that an aggregate hit rate would average away.
Its failure mode is that agreement is not accuracy: a panel that shares the same training, incentives, or blind spot can converge confidently on the wrong answer, and the method will read that shared error as high reliability. Correlated experts calibrate confidence upward exactly where they should not. The classic misuse is letting judges confer before committing — the resulting consensus looks like agreement but is really the loudest voice, and the disagreement signal is gone. The guarding discipline is to protect independence ruthlessly, to seek judges with genuinely diverse backgrounds so shared blind spots are less likely, and to remember that this method measures reliability (convergence), not validity (correctness) — it must be paired with outcome evidence wherever outcomes eventually arrive.
How it implements the components¶
bias_and_miscalibration_probe— inter-judge spread is the probe: it exposes case types where individual confidence is unreliable, independent of any single judge's self-assessment.escalation_and_override_gate— high-disagreement cases are automatically routed to a second read, panel, or adjudication, overriding the solo fast-track call.confidence_claim_format— it defines how divergence maps onto a permitted confidence claim (e.g., capping solo confidence on high-spread case types), shaping what a single judge is allowed to assert.
It does not route on a single judge's own low confidence against a fixed threshold — that absolute-confidence gate is Low-Confidence Escalation Trigger, whereas this mechanism escalates on divergence across judges even when each one feels sure. It also does not build a resolved-outcome track record or feedback_collection_loop — the outcome-based loop is Post-Outcome Recalibration Review.
Related¶
- Instantiates: Heuristic Calibration and Confidence Judgment — supplies the outcome-free, peer-divergence reliability signal.
- Sibling mechanisms: Calibration Adjustment Rule · Challenge Case Set · Confidence Bucket Review · Ecological Validity Screen · Low-Confidence Escalation Trigger · Post-Outcome Recalibration Review · Prediction Journal · Reference Class Comparison · Reliability Diagram or Calibration Curve
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Expert Disagreement Calibration operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it uses the spread among several independent experts on the same case as a live reliability signal — high disagreement caps confidence and triggers escalation.
Independent corroboration: The frozen evidence defines Expert Disagreement Calibration as 'Uses the spread among several independent experts on the same case as a live reliability signal — high disagreement caps confidence and triggers escalation', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Convergent development
Present-day reach: Multi-domain
Rationale: Using inter-expert dispersion as a reliability and uncertainty signal derives from measurement and inter-rater statistics.
Related originating lineages:
- Futurism & Strategic Foresight — Structured forecasting and Delphi practice materially developed independent forecasts, disagreement reporting, and escalation.
- Psychology — Judgment and decision research materially interprets expert overconfidence and disagreement.
Review resolution: Both reviewers agree that statistics_experimental_design is primary. I retain psychology, futurism_foresight only as formative origin lineages; convergent is appropriate because the same operational pattern arose through parallel professional lineages. Reach is multi_domain because the structure transfers across several fields but is not a near-universal human pattern, an applicability judgment kept separate from provenance. Encyclopedia synthesis is true because the exact generalized artifact is an encyclopedia-authored combination or refinement. No unresolved historical ambiguity remains after reconciling the secondary fields.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] The Delphi method structures expert judgment by having participants respond independently before any aggregation, precisely to preserve divergence and prevent the loudest or highest-status voice from manufacturing false consensus. Its independence-first discipline is what keeps a disagreement signal meaningful. ↩