Structured Expert Judgment Iteration

Iteratively elicit and refine expert judgment under uncertainty while preserving both convergence and disagreement.

Solution archetype #

1027

Problem family

Uncertainty, Evidence & Inference Failure

Problem subfamily

Evidence Fusion, Correlation & Expert Aggregation

The Diagnostic Story

Symptom: A decision depends on expert judgment, but when experts are brought together the most senior voice anchors the discussion, disagreement gets smoothed into false consensus, and the final recommendation comes out as a confident point estimate with no visible uncertainty and no record of what was contested. One-shot polling produces numbers without rationale; committee meetings produce authority-weighted conclusions that are neither auditable nor updatable.

Pivot: Elicit individual expert judgments independently before any group discussion, aggregate results and return structured feedback across rounds, and preserve both the convergence and the persistent disagreement rather than collapsing everything into a single official answer.

Resolution: Expert input becomes more reliable because it is protected from anchoring and groupthink before influence is introduced. Uncertainty and disagreement remain visible and inspectable. The decision can be made against a richer, calibrated picture of what experts actually know and where they genuinely differ.

Reach for this when you hear…

[nuclear safety assessment] “We cannot just ask them all in the same room — the senior engineer's first number will be the number everyone anchors to, and we will have wasted the expertise of six other people.”

[epidemiology] “After three rounds we could see exactly where the panel agreed and where the disagreement was irreducible — that is the uncertainty the decision-maker actually needed to see.”

[financial risk management] “One-shot expert polling gives us false precision; iterating with anonymized feedback gives us a probability distribution we can actually stress-test.”

When This Archetype Applies

Complete catalog groundingAt least one sufficient condition set is fully represented by existing primes or domain-specific abstractions.

Diagnostic problem

A decision depends on expert judgment, but unstructured expert discussion is biased by status, groupthink, anchoring, or premature consensus.

What this problem means

The structural problem is that expert judgment is valuable and fragile at the same time. Experts know things that are not yet fully measurable, but they are also subject to cognitive bias, institutional incentives, disciplinary blind spots, reputation concerns, and social pressure. When a group simply talks, the final recommendation may reflect status and momentum more than evidence.

The failure signature is an opaque “expert consensus” that hides how judgments were formed. Averages hide polarized estimates. Committee reports hide minority warnings. One-time surveys hide whether experts would revise after seeing counterarguments. Ordinary meetings hide whose assumptions shaped the room.

Show the applicability expression

Applicability expression5 distinct conditions

Incomplete evidence expertiseandBias-prone unstructured discussionandCentral estimate and uncertaintyandFeedback-improvable judgmentsandAuditable expert input

Algebraic1∧2∧3∧4∧5

groundedpartly groundedopen

5 conditions, all required.

5Required in every casenumbered 1–5

These hold no matter which pattern applies.

1

Incomplete evidence expertise · grounded

Evidence is incomplete but relevant expertise exists. more

and — also required

2

Bias-prone unstructured discussion · grounded

Unstructured discussion is likely to distort judgment. more

and — also required

3

Central estimate and uncertainty · grounded

A decision needs both central tendency and uncertainty. more

and — also required

4

Feedback-improvable judgments · grounded

Judgments can improve through controlled feedback. more

and — also required

5

Auditable expert input · grounded

Expert input must be auditable later. more

Other requirements and context (1)

Why these sit outside the expression

Supporting context — it may accompany or help interpret the situation, but it is not a load-bearing condition in a sufficient diagnostic set.

• Supporting contextPersistent disagreement is decision-relevant.

Coverage

5 of 5 conditions grounded.

Read the methodology · Download the trigger-logic data

Mechanisms / Implementations

• Delphi Study: Implements structured expert judgment through anonymous rounds, controlled feedback, and revision until useful convergence or stable disagreement is reached.
• Expert Elicitation Protocol: Defines how judgments, rationales, probabilities, confidence ranges, assumptions, and evidence claims are collected from experts.
• Anonymous Survey Round: Captures independent judgments and revisions while reducing status pressure, anchoring, and conformity.
• Structured Forecasting Panel: Uses repeated expert estimates, feedback, and uncertainty summaries to assess future events, timelines, or probabilities.
• Calibrated Probability Elicitation: Elicits ranges, probabilities, or distributions while checking for overconfidence, incoherence, and calibration problems.
• Technical Consensus Round: Iteratively refines expert positions on standards, safety thresholds, design choices, or technical interpretations without relying only on meeting-room authority.
• Judgment Aggregation Dashboard: Displays distributions, movement between rounds, confidence, subgroup variation, and unresolved disagreements so iteration remains visible.
• Rationale Coding Matrix: Organizes reasons, evidence types, assumptions, and counterarguments behind expert judgments across rounds.
• Policy Expert Panel Process: Adapts structured judgment iteration to policy questions where evidence, values, feasibility, legitimacy, and stakeholder effects interact.

Related Abstractions

Abstractions this archetype builds on — directly (a source ingredient) or as a related pattern. Links follow the typed catalog namespace.

Built directly on (3)

• Convergence: Movement toward stable state.
• Delphi Method: Expert consensus iteration.
• Uncertainty: Incomplete knowledge.

Also references 12 related abstractions

• Anchoring: Overweight initial info.
• Bayesian Updating: Update beliefs with evidence.
• Bounded Rationality: Limited decision capacity.
• Confidence Intervals: Range of plausible values.
• Confirmation Bias: Favor confirming evidence.
• Divergence Convergence
• Ensemble: Multiple comparable realizations are generated or assembled and analyzed together through a probability model and aggregation rule to characterize a distribution rather than a single trajectory.
• Epistemic Justice: Fair knowledge production.
• Groupthink: Conformity overrides realism.
• Probability: Quantifies uncertainty and likelihoods.

▸ Show 2 more

Variants

Narrower or domain-specific specializations that share this archetype's core structure. Recognized variants are established; candidate variants are provisional.

Delphi Iteration · mechanism family variant · recognized

A named variant that uses anonymous expert rounds with controlled feedback to clarify convergence, uncertainty, and disagreement.

Calibrated Expert Elicitation · implementation variant · recognized

A quantitative variant that elicits probability distributions or confidence ranges and checks calibration, overconfidence, and coherence.

Disagreement-Preserving Expert Iteration · risk or failure variant · recognized

A variant that uses expert rounds primarily to clarify persistent disagreement rather than to maximize consensus.

Technical Consensus Iteration · domain variant · candidate

A standards, engineering, or safety-oriented variant that iterates expert judgment around technical thresholds, definitions, or design interpretations.

Editorial Notes

Problem Classification

Classification: Uncertainty, Evidence & Inference Failure → Evidence Fusion, Correlation & Expert Aggregation

Problem kernel: status and group dynamics distort expert evidence aggregation

Rationale: Earliest causal condition: A decision depends on expert judgment, but unstructured expert discussion is biased by status, groupthink, anchoring, or premature consensus.

Independent corroboration: The earliest necessary condition in the frozen evidence is: A decision depends on expert judgment, but unstructured expert discussion is biased by status, groupthink, anchoring, or premature consensus. That is a evidence fusion correlation and expert aggregation problem because Multiple signals, sources, models, or experts are combined without reliability weights, independence checks, shared-lineage controls, or conflict handling.

Review outcome: Independent reviewer agreement; high confidence.