Expert Elicitation Protocol¶
Protocol — instantiates Structured Expert Judgment Iteration
Defines how judgments, rationales, probabilities, confidence ranges, assumptions, and evidence claims are collected from experts.
An Expert Elicitation Protocol is the written rulebook that fixes how judgments will be gathered before a single expert is asked anything. It is not the act of eliciting and not the study that iterates; it is the standing specification — variables and their exact definitions, the format of every answer, the sequence that keeps first judgments independent, the assumptions experts may and may not make, and the conflict disclosures required to sit on the panel. Its defining purpose is repeatability and defensibility: because the rules are written down in advance, two facilitators running the protocol get comparable inputs, and a skeptic reading the record later can see that the numbers were collected under discipline rather than improvised in a room. Where a survey round performs one collection, the protocol is the document that says how any collection must be performed.
Example¶
A national regulator commissions an expert elicitation on the annual probability of a large-magnitude seismic event affecting a proposed facility — a number that will feed a safety case, so it must withstand hostile review. Before anyone is asked, the project writes an Expert Elicitation Protocol modeled on established structured-elicitation practice[1]. It pins down the exact quantity ("annual frequency of ground motion exceeding X at the site"), forbids experts from silently importing their own siting assumptions by listing the fixed background assumptions, and specifies that each expert's initial quantiles are submitted individually and sealed before any group discussion — independence is a written rule, not a hope. It also requires every panelist to file a conflict-of-interest disclosure: prior consulting for the facility's owner, publications staking a position, funding sources.
When the elicitation runs, one panelist's disclosure reveals paid advisory work for the applicant. The protocol had a rule for exactly this: the disclosure is logged and their judgment is flagged in the record rather than quietly dropped or quietly trusted. The output is a set of individually-captured judgments whose provenance and independence a later auditor can verify line by line.
How it works¶
- Fix the frame in writing. Define each variable, its units, the answer format (point, interval, distribution), the scale, and the horizon — so "the answer" cannot drift in meaning between experts.
- Specify shared assumptions. State the conditions every expert must hold constant, closing the gap where private assumptions would otherwise make answers incomparable.
- Mandate independence sequencing. Require initial judgments to be captured privately and sealed before group exposure, as a documented step rather than a facilitator's discretion.
- Screen conflicts by rule. Require disclosures on a fixed form and specify how a disclosed conflict is handled (flag, weight, recuse) before the elicitation begins.
Tuning parameters¶
- Frame rigidity — how tightly the protocol constrains definitions and formats. Rigid frames maximize comparability but can force experts to answer a slightly wrong question precisely.
- Assumption scope — how many background conditions are fixed versus left to expert judgment. Fixing more improves comparability but risks baking a sponsor's premise into every answer.
- Disclosure threshold — what level of interest must be declared, and what a declared conflict triggers. Strict thresholds protect credibility but can thin an already-scarce expert pool.
- Independence enforcement — sealed-and-timestamped versus honor-system private answers. Stronger enforcement is auditable but heavier to administer.
When it helps, and when it misleads¶
Its strength is defensibility: a protocol turns a collection of expert opinions into a procedure that can be audited, replicated, and defended in front of a hostile reviewer, and it forecloses two silent corruptions at once — drifting definitions and undisclosed conflicts. It is what lets a safety case or a regulatory filing say "here is exactly how we asked, and who we asked."
Its failure mode is procedural rigor mistaken for substantive correctness: a beautifully specified protocol still collects garbage if the frame quietly encodes the sponsor's preferred assumption, and its very formality can launder a loaded question into an official-looking number. The classic misuse is writing the protocol to produce a defensible-looking answer rather than an honest one — precision as armor. The guarding discipline is independent review of the frame and assumptions before fielding, and a standing rule that disclosed conflicts are recorded in the output, not scrubbed from it.
How it implements the components¶
judgment_frame— its core deliverable: the written definitions, formats, scales, and assumptions every expert answers within.independent_elicitation— it mandates, as a documented and enforceable step, that first judgments are sealed before group exposure.conflict_of_interest_screen— it fixes the disclosure form and the rule for handling declared conflicts before elicitation begins.
It specifies collection but does not run the loop or score the numbers: it does not iterate rounds or report convergence (iteration_round, convergence_disagreement_report are Delphi Study's), and it does not calibration-test the experts (calibration_seed_question belongs to Calibrated Probability Elicitation).
Related¶
- Instantiates: Structured Expert Judgment Iteration — the protocol is the standing specification the archetype's rounds are run against.
- Sibling mechanisms: Anonymous Survey Round · Calibrated Probability Elicitation · Delphi Study · Judgment Aggregation Dashboard · Rationale Coding Matrix · Structured Forecasting Panel · Policy Expert Panel Process · Technical Consensus Round
Editorial Notes¶
Form Classification¶
Form family: Protocol, Workflow & Routine
Rationale: Expert Elicitation Protocol operates as a repeatable ordered procedure or handoff sequence that coordinates action because it defines how judgments, rationales, probabilities, confidence ranges, assumptions, and evidence claims are collected from experts.
Independent corroboration: The frozen evidence defines Expert Elicitation Protocol as 'Defines how judgments, rationales, probabilities, confidence ranges, assumptions, and evidence claims are collected from experts', so its operative form is Protocol, Workflow & Routine.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Operations Research
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Decision analysis and operations research formalized structured expert judgment as documented elicitation of probabilities, ranges, assumptions, and uncertainty for models and risk decisions.
Related originating lineages:
- Futurism & Strategic Foresight — Delphi and foresight methods independently developed iterative expert elicitation for uncertain futures. Formal expert-elicitation frameworks and Delphi-style structured judgment are canonical foresight practices for uncertain questions.
- Public Administration & Policy — Regulatory and safety-case use materially shaped conflict screening, documentation, and defensibility requirements.
- Statistics & Experimental Design — Statistical elicitation materially supplies calibrated probabilities, uncertainty ranges, and aggregation discipline. Probability elicitation, calibration, independence, and uncertainty ranges materially provide the measurement discipline.
Review resolution: EFSA's guidance operationalizes probability distributions and auditable elicitation. Statistics supplies calibration, while foresight and policy are major application and co-development settings.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
References¶
[1] Cooke, R. M. Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press (1991). Presents an established structured expert-judgment practice for eliciting and combining expert opinions. registry ↩