Rerun Checklist¶
Checklist — instantiates Reproducibility Protocol
Provides a lightweight confirmation list for rerunning the result path and comparing outputs against the reference.
A Rerun Checklist is the smallest possible reproducibility ritual: a short, ordered list of items a person ticks off while re-executing a result path and confirming the output still matches the reference. Its defining idea is that it operationalizes the act of checking rather than preserving anything — it is not an artifact, a document of method, or a stored bundle, but a lightweight confirmation procedure that turns "someone should verify this reproduces" into a concrete, finished-or-not task. The checklist owns no data and no code; it owns the discipline of actually running the verification and recording, item by item, that each step passed. Its value is that a rerun which is nobody's explicit job silently never happens, and a checklist makes it happen.
Example¶
A bank must regenerate a regulatory risk report each quarter, and the number that goes to the regulator has to match the certified figure exactly. The reproduction path already exists — the data pull, the calculation, the reference value from last certification — but reruns had been failing in embarrassing ways because analysts skipped steps under deadline. So the team adopts a one-page rerun checklist: confirm the input snapshot matches the certified version identifier; run the calculation; confirm the headline capital ratio equals the reference to the stated precision; confirm each subtotal is within tolerance of its reference; sign and date. On the quarter an analyst tries a shortcut, the checklist's "input identifier matches reference?" item fails at step one, catching a wrong data vintage before the report leaves the building rather than after the regulator flags it.[n1] (Figures and process are illustrative.)
How it works¶
The checklist is a bounded sequence of pass/fail confirmations tied to a known result path. Two moves make it a reproducibility mechanism and not a to-do list: each item compares an output against a stored reference (the certified figure, the benchmark table, the expected checksum), so "done" means "matched," not merely "ran"; and completing the list constitutes an explicit, attributable rerun check — a named person confirms, on the record, that the path reproduced. It presumes the materials already exist; it does not preserve the method, freeze the data, or capture the environment. It is the trigger and the tally for the verification act, deliberately thin so it is actually used.
Tuning parameters¶
- Item granularity — a handful of high-level confirmations or a fine-grained step-by-step. Coarse lists get completed but catch less; fine lists catch more but tempt rote ticking.
- Comparison strictness — exact match, tolerance band, or "looks right" per item. Strict comparisons catch drift; loose ones move faster but let small divergences slide.
- Blocking vs. advisory — whether a failed item halts the process or just logs a warning. Blocking prevents bad reruns from shipping; advisory keeps flow but relies on follow-up.
- Attribution — whether completion is signed. Signing makes the check accountable and auditable; anonymous ticking is quicker but deniable.
- Cadence trigger — what invokes the checklist (every rerun, each release, each period). Frequent triggers catch drift early; rare ones save effort but widen the blind window.
When it helps, and when it misleads¶
Its strength is leverage per gram: a one-page list, faithfully used, catches the mundane reproduction failures — wrong data vintage, skipped step, output that quietly stopped matching — that heavier machinery assumes away, and it assigns the check to a real person at a real moment. Checklists earn their keep exactly where competent people under pressure skip obvious steps.
Its failure mode is the checklist ticked without being performed — boxes checked by reflex, "matched" recorded for a comparison no one actually made. Because a completed list looks like verification, a hollow one manufactures false assurance more efficiently than having no checklist at all. The classic misuse is a compliance list signed to close a ticket. The guarding discipline is to make each item name a concrete, observable comparison whose result can itself be spot-audited — so "confirmed" points at a diff or a value, not a feeling.
How it implements the components¶
output_reference— each item compares a produced output against a stored reference value, table, or checksum, so success means the numbers match.independent_rerun_check— completing the list is the enacted rerun verification, performed and attributed to a named person on the record.
It preserves nothing itself — the frozen inputs and expected-output bundle (data_version, dependency_manifest) come from Reproducible Research Package, and the captured runtime (environment_record) from Containerized Environment Snapshot; the checklist runs and confirms against materials those siblings supply.
Related¶
- Instantiates: Reproducibility Protocol — the checklist supplies the enacted-verification strand of the archetype.
- Consumes: Reproducible Research Package — the checklist reruns and confirms the materials a research package assembles.
- Sibling mechanisms: Reproducible Research Package · Replication Package · Containerized Environment Snapshot · Workflow Script or Pipeline · Version-Controlled Analysis · Protocol Documentation · Lab Notebook Record · Decision Log · Audit Trail
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Rerun Checklist operates by compares rerun outputs with stored references through bounded pass-or-fail confirmations. That concrete deployed or enacted form is Assessment, Review & Assurance under the frozen taxonomy.
Nearest alternative: Interface, Display & Cue — Although Interface, Display & Cue can support this mechanism, the frozen evidence makes its operative form the act that compares rerun outputs with stored references through bounded pass-or-fail confirmations; the alternative is therefore secondary rather than defining.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: A lightweight repeat-and-compare checklist descends from reproducibility and replication practice.
Related originating lineages:
- Computer Science & Software Engineering — Software runbooks and regression checks materially shape executable reruns against reference outputs.
Review resolution: Both blind reviewers agree that statistics_experimental_design is the primary historical origin. Explicit reconciliation of alternate origin disagreement, origin mode disagreement, encyclopedia synthesis disagreement adopts reviewer_a's evidence: A lightweight repeat-and-compare checklist descends from reproducibility and replication practice. The selected record uses alternates=computer_science, origin_mode=cross_disciplinary_synthesis, and domain_reach=multi_domain; the other review proposed alternates=data_science, mathematics, origin_mode=single_lineage, and domain_reach=multi_domain. The selected combination better preserves the mechanism-specific formative lineages and calibrated scope; broader present-day use is not treated as proof of additional historical origin.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] The disciplined use of short confirmation checklists to prevent competent professionals from skipping obvious-but-critical steps was popularized by surgeon Atul Gawande in The Checklist Manifesto; the leverage comes from catching routine omissions, not from adding sophistication. ↩