Skip to content

Structured Expert Judgment Iteration

Iteratively elicit and refine expert judgment under uncertainty while preserving both convergence and disagreement.

The Diagnostic Story

Symptom: A decision depends on expert judgment, but when experts are brought together the most senior voice anchors the discussion, disagreement gets smoothed into false consensus, and the final recommendation comes out as a confident point estimate with no visible uncertainty and no record of what was contested. One-shot polling produces numbers without rationale; committee meetings produce authority-weighted conclusions that are neither auditable nor updatable.

Pivot: Elicit individual expert judgments independently before any group discussion, aggregate results and return structured feedback across rounds, and preserve both the convergence and the persistent disagreement rather than collapsing everything into a single official answer.

Resolution: Expert input becomes more reliable because it is protected from anchoring and groupthink before influence is introduced. Uncertainty and disagreement remain visible and inspectable. The decision can be made against a richer, calibrated picture of what experts actually know and where they genuinely differ.

Reach for this when you hear…

[nuclear safety assessment] “We cannot just ask them all in the same room — the senior engineer's first number will be the number everyone anchors to, and we will have wasted the expertise of six other people.”

[epidemiology] “After three rounds we could see exactly where the panel agreed and where the disagreement was irreducible — that is the uncertainty the decision-maker actually needed to see.”

[financial risk management] “One-shot expert polling gives us false precision; iterating with anonymized feedback gives us a probability distribution we can actually stress-test.”

Mechanisms / Implementations

  • Delphi Study: delphi_study is a method that implements the archetype by helping teams elicit, feed back, revise, aggregate, or report expert judgment.
  • Expert Elicitation Protocol: expert_elicitation_protocol is a protocol that implements the archetype by helping teams elicit, feed back, revise, aggregate, or report expert judgment.
  • Anonymous Survey Round: anonymous_survey_round is a procedure that implements the archetype by helping teams elicit, feed back, revise, aggregate, or report expert judgment.
  • Structured Forecasting Panel: structured_forecasting_panel is a workflow that implements the archetype by helping teams elicit, feed back, revise, aggregate, or report expert judgment.
  • Calibrated Probability Elicitation: calibrated_probability_elicitation is a test_or_assessment that implements the archetype by helping teams elicit, feed back, revise, aggregate, or report expert judgment.
  • Technical Consensus Round: technical_consensus_round is a ritual that implements the archetype by helping teams elicit, feed back, revise, aggregate, or report expert judgment.
  • Judgment Aggregation Dashboard: judgment_aggregation_dashboard is a metric_or_dashboard that implements the archetype by helping teams elicit, feed back, revise, aggregate, or report expert judgment.
  • Rationale Coding Matrix: rationale_coding_matrix is a artifact that implements the archetype by helping teams elicit, feed back, revise, aggregate, or report expert judgment.
  • Policy Expert Panel Process: policy_expert_panel_process is a workflow that implements the archetype by helping teams elicit, feed back, revise, aggregate, or report expert judgment.

Abstractions this archetype builds on — directly (a source ingredient) or as a related pattern. Links follow the typed catalog namespace.

Built directly on (3)

Also references 12 related abstractions

Variants

Narrower or domain-specific specializations that share this archetype's core structure. Recognized variants are established; candidate variants are provisional.

Delphi Iteration · mechanism family variant · recognized

A named variant that uses anonymous expert rounds with controlled feedback to clarify convergence, uncertainty, and disagreement.

Calibrated Expert Elicitation · implementation variant · recognized

A quantitative variant that elicits probability distributions or confidence ranges and checks calibration, overconfidence, and coherence.

Disagreement-Preserving Expert Iteration · risk or failure variant · recognized

A variant that uses expert rounds primarily to clarify persistent disagreement rather than to maximize consensus.

Technical Consensus Iteration · domain variant · candidate

A standards, engineering, or safety-oriented variant that iterates expert judgment around technical thresholds, definitions, or design interpretations.