Assumption Audit Checklist¶
Audit checklist — instantiates Assumption-Light Inference
Enumerates the assumptions a planned inference rests on and flags which ones would change the conclusion if they failed — before any test is run.
Most fragile inferences fail not because someone chose the wrong test but because nobody ever wrote down what the test was quietly assuming. Assumption Audit Checklist is the paper-first artifact that fixes that: a standing list of the assumptions a planned analysis will lean on — distributional form, independence, stationarity, measurement scale, sampling frame, missingness, outlier handling, and any domain-specific commitments — filled in before the data are touched. Its defining move is not to test anything but to make the burden explicit and then sort it: separating the assumptions whose failure would flip the conclusion from the ones that are harmless simplifications. It is a planning document, not a diagnostic; its whole value is that method selection stops being a reflex and becomes a response to a named threat.
Example¶
A clinical-research team is about to compare recovery times between two post-surgical protocols and is drafting a pre-registered statistical analysis plan. Before choosing a test, the lead runs the audit checklist. Row by row, they write down what a two-sample t-test would assume: recovery times are roughly normal (they are not — the distribution is right-skewed, with a long tail of slow recoverers), variances are equal across arms (unknown), observations are independent (mostly, but some patients share a surgeon), and the outcome is on a true interval scale (it is, in days). Each row gets a second column — would the conclusion change if this failed? Normality and the skew score high; the surgeon-clustering scores medium; the interval-scale assumption scores low because days really are days.
The output is not a result but a prioritized ledger: two consequential assumptions to design around, one to note, one to ignore. That ledger is what tells the team to pre-specify a rank-based comparison and a robust summary rather than defaulting to the mean — and, because it is on paper before the unblinding, it is what keeps the eventual method choice from looking like a post-hoc reaction to an inconvenient p-value.
How it works¶
The checklist is a two-column instrument. The first column enumerates; the second adjudicates.
- Enumerate exhaustively. Walk the planned procedure and write every assumption it makes, including the ones that feel too obvious to state — those are the ones that bite.
- Score for consequence. For each, ask a single counterfactual: if this assumption is false to the degree plausible here, does the claim, action, or risk judgment change? High/medium/low is enough; precision is not the point.
- Draw the boundary. The high-consequence rows become the design agenda — the assumptions the method choice, the sensitivity work, and the interpretation limit must all answer to. The low rows are explicitly parked so effort is not wasted defending them.
- Freeze and date it. The artifact is time-stamped ahead of analysis so the audit cannot be quietly rewritten to match the answer.
Tuning parameters¶
- Granularity — one line per assumption family versus a fully decomposed row per sub-assumption. Finer granularity catches hidden commitments but bloats the ledger and invites checkbox fatigue.
- Consequence scale — coarse (high/medium/low) versus a scored rubric tied to decision stakes. Finer scoring aids triage on high-stakes work but adds false precision on routine analyses.
- Bindingness — advisory notes versus a pre-registered plan that locks the method to the audit. Locking buys credibility at the cost of flexibility when the data surprise you.
- Scope — analysis assumptions only, or also sampling, measurement, and causal-identification commitments. Wider scope prevents an assumption-light method from masking a deeper validity problem, but risks duplicating neighboring archetypes' work.
When it helps, and when it misleads¶
Its strength is that it kills the most common failure of the whole archetype at the root: choosing a familiar method by habit and only later discovering its assumptions never held. By forcing enumeration before selection, it converts researcher degrees of freedom — the many undocumented choices between data and result — from an invisible risk into a governed list.[1] It also gives two disagreeing analysts a shared object: they can argue about which row is consequential instead of talking past each other about whether the model is "valid."
Its failure mode is theater. A checklist filled in to be filed rather than to be used produces a false sense of rigor — every box ticked, no assumption actually relaxed. The classic misuse is retro-fitting: writing the audit after the analysis so it ratifies the method already chosen. It is also only a map, never a measurement — it says which assumptions matter but cannot tell you whether they hold; that requires looking at the data. The guarding discipline is to date the audit before analysis and to require that every high-consequence row visibly changes a downstream decision (method, sensitivity check, or stated limit) — an audit that changes nothing was decorative.
How it implements the components¶
assumption_audit— this mechanism is the audit: the exhaustive enumeration of the assumptions behind the planned inference is its primary deliverable.consequential_assumption_boundary— the second column draws the boundary, separating conclusion-changing assumptions from harmless simplifications so effort concentrates where failure would matter.
It stops at naming and prioritizing assumptions on paper. It does NOT implement evidence_scale_alignment or assumption_review_trigger — those belong to Diagnostic Plot Review, which checks the same listed assumptions against the fitted data and reopens the audit when a graphic contradicts a row.
Related¶
- Instantiates: Assumption-Light Inference — supplies the named assumption threats every other mechanism responds to.
- Sibling mechanisms: Diagnostic Plot Review · Model Comparison Table · Median-Based Summaries · Bootstrap-Like Checks · Nonparametric Tests · Permutation Tests · Rank-Based Methods · Robust Statistics
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Enumerates the assumptions a planned inference rests on and flags which ones would change the conclusion if they failed — before any test is run, making its operative form a bounded evaluation of existing evidence or work that produces a finding or disposition.
Independent corroboration: The frozen evidence defines Assumption Audit Checklist as 'Enumerates the assumptions a planned inference rests on and flags which ones would change the conclusion if they failed — before any test is run', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Enumerating conclusion-changing assumptions before a planned inference is a standard statistical model and study-design discipline.
Related originating lineages:
- Data Science & Analytics — Model-validation checklists operationalize assumptions in analysis pipelines.
- Philosophy — Logic and epistemology distinguish premises from warranted conclusions.
Review resolution: Statistical experimental design is the agreed primary because the checklist precedes model choice and pre-registration. Data-science pipeline assumptions and philosophical warrant analysis materially form the generalized artifact; medicine and engineering are high-stakes applications rather than separate origins. The consequence-boundary checklist is an Encyclopedia synthesis.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
The checklist is deliberately upstream of every other mechanism here: it names the threats, but relaxing them is someone else's job. Keeping the audit separate from the fix is what lets a team improve its threat list without re-litigating which test to run.
References¶
[1] Researcher degrees of freedom — the many undocumented analytic choices (exclusions, transformations, model form, stopping rules) that, taken together, let an analyst find a "significant" result almost anywhere. Named by Simmons, Nelson, and Simonsohn (2011); pre-specifying those choices is the standard corrective, which is exactly what a dated audit checklist does. withdrawn registry ↩