Domain Expert Calibration Panel¶
Elicitation panel — instantiates Metric-Space Specification and Validation
Convenes domain experts to judge which pairs are genuinely near or far, calibrating the metric's semantics against human expertise.
A distance function can be formally perfect and still mean the wrong thing. Domain Expert Calibration Panel brings the people who understand the domain into the loop before the metric is trusted, asking them to judge concrete pairs — are these two things genuinely similar, and is this one closer to that one than to a third? Its defining move is that the ground truth is produced by human judgment, not assumed from labels lying around: experts supply the semantic anchor against which any numerical distance is checked. The panel establishes what "near" is supposed to mean in this domain and manufactures a set of judged cases the metric must honor. It is an elicitation instrument, not an automated scorer.
Example¶
A hospital is building a case-similarity tool so that when a clinician opens a patient's chart, the system surfaces the most comparable past episodes. Before deploying any distance function over the episode features, the data team convenes three attending physicians. They present a curated deck of episode pairs — some the engineers expect to be near, some far, and a deliberate set of tricky borderline pairs — and ask each physician, independently, to rate similarity and to say why. The physicians agree that two chest-pain episodes with different underlying causes are clinically far despite near-identical vitals, and that a case with sparse data but a matching diagnosis is near despite few shared features.
Those judgments become two things at once. First, a written proximity contract: in this domain, closeness means comparable clinical trajectory and cause, not feature-vector overlap. Second, a labeled bank of judged pairs — including the physicians' disagreements, which mark genuinely ambiguous territory — that any candidate metric must reproduce before it is allowed near a clinician. The panel has not scored a metric; it has defined the target a metric will later be scored against.
How it works¶
- Sample cases to span the semantic space, not the frequency distribution — deliberately including near pairs, far pairs, and the borderline and adversarial pairs where intuition and formula are most likely to diverge.
- Elicit judgments independently first, then reconcile. Experts rate blind to each other and to the metric's own output, so the panel captures genuine domain intuition rather than anchoring on a number or on the loudest voice.
- Measure agreement among the experts. Where they concur, the target is firm; where they diverge, that disagreement is itself data marking irreducibly ambiguous cases the metric should not be expected to resolve.
- Write the semantics down. The panel's output is an explicit contract — this is what near and far mean here — plus the judged case bank, both handed to downstream evaluation.
Tuning parameters¶
- Panel composition and size — how many experts and how diverse. More and more varied experts widen coverage and expose real disagreement, but raise cost and the effort of reconciliation.
- Case-sampling strategy — random, stratified across the domain, or adversarially concentrated on borderline pairs. Adversarial sampling stresses the metric hardest but is not representative of everyday use.
- Judgment format — absolute similarity ratings, pairwise "which is closer" comparisons, or free grouping. Relative comparisons are usually more reliable than absolute numeric ratings.
- Blinding discipline — whether experts see the metric's output before judging. Showing it is faster but risks automation bias contaminating the ground truth.
When it helps, and when it misleads¶
Its strength is that it is the only mechanism here that can catch semantic mismatch — a metric that is numerically self-consistent yet clinically, legally, or physically nonsensical. Automated checks presuppose a correct label set; the panel is where that label set, and the meaning behind it, is created. It also surfaces where experts genuinely disagree, which honestly bounds what any metric can be expected to do.
Its failure mode is that expert judgment is itself noisy[1] and biased: panels can share blind spots, defer to seniority, or rate inconsistently, and inter-rater reliability is often lower than teams assume. A small or homogeneous panel can enshrine a confident consensus that is simply wrong, and the metric then inherits that error wearing the authority of "the experts said so." The classic misuse is treating a single unmeasured panel as oracular ground truth. The guarding discipline is to quantify inter-rater agreement, keep the panel diverse, and treat low-agreement cases as ambiguous rather than forcing a verdict — so the metric is calibrated against reliable judgments and honestly excused on the unreliable ones.
How it implements the components¶
domain_element_set— the panel forces an explicit answer to what exactly are we comparing (episodes, not patients or claims), pinning down the domain over which distance will be judged.proximity_semantics_contract— its central output is the written statement of what near and far mean in this domain, derived directly from how experts reasoned about the pairs.validation_case_set— it produces the bank of expert-judged pairs, including borderline and disagreement cases, that downstream evaluation tests the metric against.
It does not compute distances or score a candidate function's accuracy against the judged set — that automated evaluation is Nearest-Neighbor Benchmark, its nearest twin; the panel manufactures the ground truth, the benchmark consumes it. Nor does it set the action neighborhood_threshold_policy — that is Distance Threshold Review.
Related¶
- Instantiates: Metric-Space Specification and Validation — it supplies the human-grounded semantic target the archetype validates every candidate metric against.
- Sibling mechanisms: Nearest-Neighbor Benchmark · Distance-Choice Sensitivity Analysis · Distance Threshold Review · Metric Axiom Test Suite · Pairwise Distance Matrix · Feature Scaling and Normalization Pipeline
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: The panel independently judges sampled pairs, measures expert agreement, and produces calibrated findings about which semantic distances the metric should or should not resolve.
Nearest alternative: Communication, Facilitation & Learning — Reconciliation is participatory, but the panel's defining output is an evidence-based validation target and ambiguity finding for the metric, not shared learning alone.
Review outcome: Adjudicated after independent review; medium confidence.
Origin Attribution¶
Primary origin: Data Science & Analytics
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Metric-learning practice established eliciting expert near-far judgments as semantic anchors for learned or chosen distance functions.
Related originating lineages:
- Ethnography & Qualitative Methods — Structured expert elicitation supplied facilitated interpretation of disagreements and boundary cases.
- Statistics & Experimental Design — Inter-rater and psychometric methods supplied independent judgment, agreement measurement, and calibration diagnostics.
Review resolution: The mechanism first coheres as data-scientific metric learning from expert similar/dissimilar judgments, synthesized with statistical reliability and qualitative elicitation rather than originating as a generic psychometric panel.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
References¶
[1] Kahneman, D., Sibony, O., and Sunstein, C. R. Noise: A Flaw in Human Judgment. Little, Brown Spark (2021). Describes professional judgment as vulnerable to both bias and unwanted inconsistency across decisions that should be identical. registry ↩