Signal Scoring Rubric¶
Test or assessment — instantiates Weak Signal Triage
Separates plausibility, impact, evidence direction, and response cost so ambiguous signals are not flattened into one misleading score.
Signal Scoring Rubric scores an ambiguous signal on several separate axes — how plausible it is, how much would be at stake if it were real, which way its evidence is currently moving, and how costly the cheapest informative response would be — and it deliberately refuses to collapse them into a single number. Its defining move is dimensional separation: a low-plausibility / high-impact signal and a low-plausibility / low-impact signal must read differently, and a rubric that averages them into one score destroys exactly the distinction triage exists to preserve. The rubric produces a profile, not a rank. It answers "what shape is this signal?" — not "what should we do about it?" (that is the board's job) and not "what threshold trips escalation?" (that is the playbook's). Its value is that it keeps confidence and consequence from bleeding into each other before anyone decides anything.
Example¶
A regional health department's epidemiology unit receives a report of five cases of an unusual respiratory presentation in one rural county over ten days. Someone wants a single "how worried should we be, 1–10?" The rubric refuses. It scores four axes separately. Plausibility: could be a reporting artifact or a coincidental cluster — moderate, with the reasoning written down. Impact: if this were a novel transmissible pathogen, severe — high. Evidence trajectory: only one time-point so far, so direction is unknown — flat. Response cost: a targeted data pull and three clinician phone calls are cheap and fully reversible — low.
Read as a profile, the shape is unmistakable and actionable: low-confidence, high-impact, cheap to probe — precisely the combination that justifies a bounded probe rather than either dismissal or alarm. Had the four axes been averaged, the moderate plausibility would have dragged the score to an unremarkable middle and the severe-if-real impact would have vanished into it. The separation is the information.
How it works¶
- Fixed axes with anchored scales. Plausibility, impact, evidence direction, and response cost each have a small, defined scale with concrete anchors ("high impact" means this kind of consequence, with examples).
- Rationale per axis. Each score carries a one-line reason, so the number is a compressed argument, not a verdict handed down.
- Never a weighted sum. The rubric outputs the vector; it does not roll the axes into a composite, because the composite is what hides the tradeoff.
- Confidence held apart from consequence. Plausibility and impact are scored independently by design, so a severe-but-uncertain signal keeps both properties visible.
Tuning parameters¶
- Axis set — which dimensions are scored. More axes capture more nuance but slow scoring and dilute attention; too few re-collapse the very distinctions the rubric protects.
- Scale granularity — three bands vs. a ten-point scale per axis. Finer scales feel precise but invite false precision over what are judgment calls.
- Anchor specificity — how concretely each level is defined. Sharp anchors improve consistency across scorers; vague ones let the same signal score differently depending on who holds the pen.
- Worst-plausible-impact cell — whether to surface an explicit "if wrong in the severe direction" reading. Including it protects high-impact/low-confidence signals from being averaged away, at the cost of some alarm.
When it helps, and when it misleads¶
Its strength is preserving the archetype's fourth invariant — the separation of confidence and consequence. By keeping plausibility and impact on different axes, it stops a merely-uncertain signal from being dismissed and stops a merely-scary one from being over-believed, and it makes the tradeoff a decision-maker faces explicit rather than buried.
Its failure mode is pseudo-precision and re-collapse. A rubric with numbers on it invites people to average anyway, or to treat a scored judgment as an objective measurement — the classic misuse of any scoring instrument, and the reason a single risk score can actively mislead by hiding whether the danger is a false positive or a false negative.[n1] The mirror failure is axis proliferation into analysis paralysis, where scoring becomes the work. The guarding discipline is to keep the axes few and hard-anchored, require a written rationale on each, and forbid a composite — the profile must be carried forward whole into whoever decides the response.
How it implements the components¶
Signal Scoring Rubric fills the assessment side of the archetype — producing the separated judgments the rest of triage reasons from:
plausibility_assessment— scores whether the signal could reflect a real change versus coincidence or artifact, with rationale and uncertainty attached.impact_estimate— scores the consequence-if-real, independently of how likely it is.evidence_trajectory— scores the current direction of evidence (accumulating, flat, fading) as its own axis.response_cost_assessment— scores the cost and reversibility of the cheapest informative response, so action can be calibrated to evidence strength.
It does not turn these scores into an assigned response_tier or a named signal_owner — that is the Emerging Issue Triage Board; and it does not set the escalation_boundary at which scores trigger a prepared response — that is the Escalation Playbook. The rubric describes the signal; others decide and act.
Related¶
- Instantiates: Weak Signal Triage — the rubric supplies the separated assessment the board tiers on and the playbook thresholds against.
- Consumes: Weak Signal Log supplies the recorded observation the rubric scores.
- Sibling mechanisms: Weak Signal Log · Emerging Issue Triage Board · Watchlist Review · Probe Experiment · Early Warning Forum · Uncertainty Dashboard · Escalation Playbook
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Signal Scoring Rubric operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it separates plausibility, impact, evidence direction, and response cost so ambiguous signals are not flattened into one misleading score.
Independent corroboration: The frozen evidence defines Signal Scoring Rubric as 'Separates plausibility, impact, evidence direction, and response cost so ambiguous signals are not flattened into one misleading score', so its operative form is Assessment, Review & Assurance.
Nearest alternative: Interface, Display & Cue — Signal Scoring Rubric includes features of a user-facing prompt, display, template, or perceptual cue that shapes attention and action at the point of use, but its defining operation is a bounded evaluation of existing evidence or work that produces a finding or disposition.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Operations Research
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Universal
Rationale: Separating plausibility, impact, evidence direction, and response cost is a multi-criteria decision-analysis structure rather than a single statistical estimate. DOE's MCDA guidance formalizes distinct criteria and weights; statistics supplies calibration of evidence.
Related originating lineages:
- Data Science & Analytics — Data science, analytics, and operational monitoring supplies a parallel or contributing lineage for the mechanism's defining operation: separates plausibility, impact, evidence direction, and response cost so ambiguous signals are not flattened into one misleading score.
- Economics & Finance — economics_finance contributes economics, finance, and mechanism-design practice to this mechanism's defining operation—Separates plausibility, impact, evidence direction, and response cost so ambiguous signals are not flattened into one misleading score—without displacing the selected primary historical lineage.
- Mathematics — Mathematical modeling, proof, and abstract-structure practice supplies a parallel or contributing lineage for the mechanism's defining operation: separates plausibility, impact, evidence direction, and response cost so ambiguous signals are not flattened into one misleading score.
- Organizational & Management Science — Rubrics make cross-reviewer judgments comparable and auditable.
- Security Studies & Intelligence Analysis — Indicators are assessed by credibility, consequence, and action cost under ambiguity.
- Statistics & Experimental Design — Evidence direction and confidence require distinct statistical treatment.
Review resolution: The blind reviewers disagree on primary lineage (operations_research versus statistics_experimental_design). Authoritative or primary research supports operations_research as the best historical origin: Separating plausibility, impact, evidence direction, and response cost is a multi-criteria decision-analysis structure rather than a single statistical estimate. DOE's MCDA guidance formalizes distinct criteria and weights; statistics supplies calibration of evidence. The cited U.S. Department of Energy, Multi-Criteria Decision Analysis directly supports the mechanism's defining operation. All independently supported contributing domains are retained without an arbitrary cap. origin_mode=cross_disciplinary_synthesis records lineage, while domain_reach=universal records later applicability separately from provenance.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
Notes¶
[n1] Type I / Type II errors — the distinction between a false positive (acting on a signal that was noise) and a false negative (dismissing a signal that was real). A single collapsed score obscures which error a decision is risking; scoring plausibility and impact on separate axes keeps both error directions in view, which is the whole point of not averaging. ↩