The preregistration revolution.¶
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600-2606.
Cited by¶
8 citations across 8 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Experimental Design
- Pre-registration and open science practices (sharing data, code, and analysis plans) have become increasingly common, driven by concerns about reproducibility and publication bias (the tendency to publish positive results and suppress negative results), as Nosek et al. (2018) describe in their account of the preregistration revolution.
This sourcedescribes the rise of pre-registration and open science as institutional responses to publication bias and the reproducibility crisis, distinguishing prediction-testing from postdiction.
- Pre-registration and open science practices (sharing data, code, and analysis plans) have become increasingly common, driven by concerns about reproducibility and publication bias (the tendency to publish positive results and suppress negative results), as Nosek et al. (2018) describe in their account of the preregistration revolution.
- Holdout Set
- Product experimentation: always-on holdout cells estimating long-run treatment effects; the never-treated cohort in growth and personalisation systems. Audit and forensics: reserved samples the auditee does not know are under scrutiny; rotating sub-samples; sealed evidentiary piles. Software engineering: an acceptance test suite the developer may not consult while refactoring; canary traffic reserved from a new build. Forecasting and science: locked questions resolving later; replication cohorts in GWAS; registered reports reviewed before data are seen.
This sourceArgues pre-registration and registered reports separate hypothesis-generating from hypothesis-testing evidence by forward-blinding the analyst — the temporal form of holdout.
- Product experimentation: always-on holdout cells estimating long-run treatment effects; the never-treated cohort in growth and personalisation systems. Audit and forensics: reserved samples the auditee does not know are under scrutiny; rotating sub-samples; sealed evidentiary piles. Software engineering: an acceptance test suite the developer may not consult while refactoring; canary traffic reserved from a new build. Forecasting and science: locked questions resolving later; replication cohorts in GWAS; registered reports reviewed before data are seen.
- Informal Fallacy
- a result fished from the data is dressed as a prediction it was reverse-engineered to fit, a content defect (the hypothesis is not independently warranted of the result) that no check of the statistical form exposes, and methodological norms like pre-registration exist partly to forbid it.
This sourceArgues preregistration fixes the hypothesis in advance, structurally forbidding HARKing and p-hacking-narrative post-hoc rationalization.
- a result fished from the data is dressed as a prediction it was reverse-engineered to fit, a content defect (the hypothesis is not independently warranted of the result) that no check of the statistical form exposes, and methodological norms like pre-registration exist partly to forbid it.
- Journalistic Objectivity
- Scientific peer review and replication. Blinded review, editor-selected reviewers without financial stake, pre-registration binding authors before outcomes are known, and independent replication as external verification.
This sourceArgues pre-registration binds researchers' hypotheses and analyses before outcomes are known, separating prediction from postdiction — the detached-verification structure (a pre-commitment that the interested author cannot reach through) applied to peer review and replication.
- Scientific peer review and replication. Blinded review, editor-selected reviewers without financial stake, pre-registration binding authors before outcomes are known, and independent replication as external verification.
- Motivated Reasoning
- The countermeasure family transfers with the diagnosis, since blinding, pre-commitment, and adversarial scrutiny are defined over stages of a pipeline rather than over minds.
This sourceFrames pre-commitment as a countermeasure defined over pipeline stages rather than over individual minds, transferable across research settings.
- The countermeasure family transfers with the diagnosis, since blinding, pre-commitment, and adversarial scrutiny are defined over stages of a pipeline rather than over minds.
- Unfalsifiability
- In each of these settings the practical question is identical and needs no technical vocabulary: before this result came in, what result would have counted against the claim, and who was entitled to say so.
This sourceArgues that defining the research questions and the analysis plan before the outcomes are observed is what separates a tested prediction from a postdiction generated after the result is known.
- In each of these settings the practical question is identical and needs no technical vocabulary: before this result came in, what result would have counted against the claim, and who was entitled to say so.
- Unreliable Narrator
- The independence condition is exercised through pre-registration, which breaks the distortion by making the account independent of the result.
This sourceEstablishes that preregistration breaks publication-bias distortion by fixing the analysis plan and securing acceptance independent of the result, making the published account independent of the outcome.
- The independence condition is exercised through pre-registration, which breaks the distortion by making the account independent of the result.
Mechanisms¶
- Claim Scope Freeze
- Freeze too late and the "before" already reflects the coming shift; press too hard for precision and you record a crisp criterion the speaker never actually held, then unfairly hold them to it — a form of moving-the-goalposts run in reverse.
This sourceExplains that a claim must be frozen before relevant outcomes are observed if prediction is to remain distinguishable from postdiction.
- Freeze too late and the "before" already reflects the coming shift; press too hard for precision and you record a crisp criterion the speaker never actually held, then unfairly hold them to it — a form of moving-the-goalposts run in reverse.
Verification¶
This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.
Registry ID ref:ad3c5dcfb535 · see in the full table