Evidence Provenance Checklist¶
Checklist — instantiates Cascade Initiation Bias Diagnosis and Correction
A checklist for classifying whether cited reasons are primary, secondary, independent, current, relevant, and verified.
A pile of "supporting evidence" looks convincing precisely because it is undifferentiated — twelve citations weigh more than one until you ask what kind of thing each citation is. Evidence Provenance Checklist is the per-item rubric that forces that question. It scores every cited reason on a fixed set of axes — primary or secondary, independent or derived, current or stale, relevant or tangential, verified or unverified — and stamps each with a confidence annotation. The one idea that makes it this mechanism is that it grades sources one at a time against a uniform rubric, before anyone is allowed to add them up. It does not trace how citations connect to each other; it classifies each in isolation so the aggregate can no longer hide behind its own bulk.
Example¶
A clinical guideline recommends a supplement "shown to reduce risk," backed by twelve references, and reviewers are inclined to accept it as well-established. The checklist is run against each reference in turn. Nine turn out to be secondary — review articles and commentaries citing other work. Two are the same primary randomized trial, cited twice. One is a mechanistic cell-study only tangentially relevant to the clinical claim. Scored on the rubric, the tally is unambiguous: one relevant, independent, verified primary trial; everything else is derived, duplicated, or off-target. The "twelve studies" do not vanish, but their provenance is now annotated, and the recommendation's true evidentiary base — a single trial — is legible on the page instead of buried in a reference count.
How it works¶
- Apply fixed axes uniformly — every source is scored on the same columns (primary/secondary, independent/derived, current/stale, relevant/tangential, verified/unverified), so nothing is graded on vibes.
- Score in isolation, before aggregating — rate each item on its own to stop a prestigious source from haloing its neighbors.
- Gate on the conjunction — an item counts as genuinely evidence-bearing only if it clears primary AND independent AND relevant AND verified; the gate is the test, not any single column.
- Emit a per-claim confidence annotation — roll the item scores into an explicit rating of how much real evidence the claim actually rests on.
Tuning parameters¶
- Axis set — which columns the rubric carries. Adding axes (e.g. conflict-of-interest, sample size) catches more but slows scoring and invites checkbox fatigue.
- Scoring granularity — binary flags versus graded scales. Grades capture nuance but blur the crisp "is this evidence-bearing or not" gate.
- Rater design — single scorer versus independent double-rating with reconciliation. Double-rating catches errors but doubles cost.
- Prestige-blinding — whether author, journal, and institution are hidden during scoring. Blinding fights the halo but can strip context that legitimately bears on quality.
- Aggregation rule — how item scores combine into the claim annotation (weakest-link, weighted, or count of gate-passers).
When it helps, and when it misleads¶
Its strength is that it makes provenance explicit and comparable, and it catches the most common inflation move directly: a wall of secondary and duplicated citations dressed up as many independent confirmations.
Its failure mode is that a checklist rates the form of a source, not its truth — a perfectly primary, independent, verified study can still be wrong, and a filled-in rubric radiates a false confidence exactly when every box is ticked. This is the well-known hazard of turning judgment into a checklist: the ritual of completion substitutes for the thinking it was meant to scaffold, a risk that frameworks like GRADE are careful to frame as structured judgment, not a mechanical score.[n1] The classic misuse is scoring each item's provenance while staying blind to whether the items are truly independent of one another — passing nine derived citations because each individually "looks primary." The guarding discipline is to keep the independence axis load-bearing and to cross-check it against a citation trace before trusting the count.
How it implements the components¶
source_confidence_annotation— its output is exactly this: a per-source, per-claim confidence-and-status annotation, drawn from the rubric scores.assumed_information_advantage_test— the primary/independent/relevant/verified gate tests, item by item, whether a cited reason actually carries the evidentiary advantage it is credited with.
It does not follow citations backward to find where they converge — that is Common Source Citation Check's source_chain_reconstruction — and it does not elicit actors' unstated private signals, which is Private Signal Survey's private_signal_visibility_buffer.
Related¶
- Instantiates: Cascade Initiation Bias Diagnosis and Correction — converts an undifferentiated citation stack into graded, annotated provenance.
- Sibling mechanisms: Decision Sequence Timeline · Initiator Interview Protocol · Private Signal Survey · Blind Independent Vote Reset · Source Disclosure Brief · Network Position Review · Common Source Citation Check
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Evidence Provenance Checklist operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it a checklist for classifying whether cited reasons are primary, secondary, independent, current, relevant, and verified.
Independent corroboration: The frozen evidence defines Evidence Provenance Checklist as 'A checklist for classifying whether cited reasons are primary, secondary, independent, current, relevant, and verified', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Library & Information Science
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Universal
Rationale: Source provenance, currency, independence, and verification are core bibliographic and information-evaluation concerns.
Related originating lineages:
- Criminology & Forensic Studies — Forensic chain-of-custody practice independently formalized verification of origin, handling, and authenticity.
- Security Studies & Intelligence Analysis — Intelligence source evaluation materially developed systematic grading of source reliability, corroboration, relevance, and recency.
Review resolution: Both reviewers agree that library_information_science is primary. I retain security_intelligence, criminology_forensic only as formative origin lineages; cross_disciplinary_synthesis is appropriate because the final form materially combines the agreed primary with the retained formative lineages. Reach is universal because the structure is portable across essentially any domain with the stated problem, an applicability judgment kept separate from provenance. Encyclopedia synthesis is true because the exact generalized artifact is an encyclopedia-authored combination or refinement. No unresolved historical ambiguity remains after reconciling the secondary fields.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; medium confidence.
Notes¶
[n1] GRADE (Grading of Recommendations Assessment, Development and Evaluation) — a widely adopted framework for rating the certainty of evidence behind clinical recommendations across dimensions such as risk of bias, directness, consistency, and precision; it is explicitly designed to make evidence quality transparent while insisting the ratings inform judgment rather than replace it. ↩