The preregistration revolution.¶
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600-2606.
Cited by¶
11 citations across 11 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Experimental Design
- Pre-registration and open science practices (sharing data, code, and analysis plans) have become increasingly common, driven by concerns about reproducibility and publication bias (the tendency to publish positive results and suppress negative results), as Nosek et al. (2018) describe in their account of the preregistration revolution.
This sourceDescribes the rise of pre-registration and open science as institutional responses to publication bias and the reproducibility crisis, distinguishing prediction-testing from postdiction.
Supported in partVerified against the work's full text
Nosek et al. document a rapid rise in preregistration and growing incentives for reproducibility, with open-data badges raising data sharing; code sharing and a causal 'driven by' account are not given.
“Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice.”
- Pre-registration and open science practices (sharing data, code, and analysis plans) have become increasingly common, driven by concerns about reproducibility and publication bias (the tendency to publish positive results and suppress negative results), as Nosek et al. (2018) describe in their account of the preregistration revolution.
- Holdout Set
- Product experimentation: always-on holdout cells estimating long-run treatment effects; the never-treated cohort in growth and personalisation systems. Audit and forensics: reserved samples the auditee does not know are under scrutiny; rotating sub-samples; sealed evidentiary piles. Software engineering: an acceptance test suite the developer may not consult while refactoring; canary traffic reserved from a new build. Forecasting and science: locked questions resolving later; replication cohorts in GWAS; registered reports reviewed before data are seen.
This sourceArgues pre-registration and registered reports separate hypothesis-generating from hypothesis-testing evidence by forward-blinding the analyst — the temporal form of holdout.
Supported in partVerified against the work's full text
The paper supports only the registered-reports item (review before outcomes are observed); product, audit, software and GWAS/forecasting items are not covered.
“With Registered Reports, authors submit their research question and methodology to the journal for peer review before observing the outcomes of the research.”
- Product experimentation: always-on holdout cells estimating long-run treatment effects; the never-treated cohort in growth and personalisation systems. Audit and forensics: reserved samples the auditee does not know are under scrutiny; rotating sub-samples; sealed evidentiary piles. Software engineering: an acceptance test suite the developer may not consult while refactoring; canary traffic reserved from a new build. Forecasting and science: locked questions resolving later; replication cohorts in GWAS; registered reports reviewed before data are seen.
- Informal Fallacy
- a result fished from the data is dressed as a prediction it was reverse-engineered to fit, a content defect (the hypothesis is not independently warranted of the result) that no check of the statistical form exposes, and methodological norms like pre-registration exist partly to forbid it.
This sourceArgues preregistration fixes the hypothesis in advance, structurally forbidding HARKing and p-hacking-narrative post-hoc rationalization.
Supported in partVerified against the work's full text
Nosek et al. support the dressing-postdiction-as-prediction problem and preregistration as its remedy, but frame preregistration as labelling prediction versus postdiction, not forbidding it.
“dressing up discovery as tests of theoretical predictions undermines the credibility of all science by making it impossible to distinguish hypothesis generation from hypothesis testing”
- a result fished from the data is dressed as a prediction it was reverse-engineered to fit, a content defect (the hypothesis is not independently warranted of the result) that no check of the statistical form exposes, and methodological norms like pre-registration exist partly to forbid it.
- Journalistic Objectivity
- Scientific peer review and replication. Blinded review, editor-selected reviewers without financial stake, pre-registration binding authors before outcomes are known, and independent replication as external verification.
This sourceArgues pre-registration binds researchers' hypotheses and analyses before outcomes are known, separating prediction from postdiction — the detached-verification structure (a pre-commitment that the interested author cannot reach through) applied to peer review and replication.
Supported in partVerified against the work's full text
Nosek et al. support pre-registration fixing hypotheses and analyses before outcomes are known; they do not cover blinded review or financially disinterested reviewers.
“An effective solution is to define the research questions and analysis plan before observing the research outcomes—a process called preregistration.”
- Scientific peer review and replication. Blinded review, editor-selected reviewers without financial stake, pre-registration binding authors before outcomes are known, and independent replication as external verification.
- Motivated Reasoning
- The countermeasure family transfers with the diagnosis, since blinding, pre-commitment, and adversarial scrutiny are defined over stages of a pipeline rather than over minds.
This sourceFrames pre-commitment as a countermeasure defined over pipeline stages rather than over individual minds, transferable across research settings.
Supported in partVerified against the work's full text
Nosek et al. support preregistration and blinding as stage-based safeguards in the research pipeline, not the claimed transfer or adversarial scrutiny.
“In prediction, the problem of forking paths is avoided because the analytic pipeline is specified before observing the data.”
- The countermeasure family transfers with the diagnosis, since blinding, pre-commitment, and adversarial scrutiny are defined over stages of a pipeline rather than over minds.
- Unfalsifiability
- In each of these settings the practical question is identical and needs no technical vocabulary: before this result came in, what result would have counted against the claim, and who was entitled to say so.
This sourceArgues that defining the research questions and the analysis plan before the outcomes are observed is what separates a tested prediction from a postdiction generated after the result is known.
Supported in partVerified against the work's full text
Supports the pre-specification half: stating the prediction and analysis plan before outcomes are seen is what keeps results from being rationalised afterward.
“A common case is when the researcher’s prediction is vague so that many possible outcomes can be rationalized after the fact as supporting the prediction.”
- In each of these settings the practical question is identical and needs no technical vocabulary: before this result came in, what result would have counted against the claim, and who was entitled to say so.
- Unreliable Narrator
- The independence condition is exercised through pre-registration, which breaks the distortion by making the account independent of the result.
This sourceEstablishes that preregistration breaks publication-bias distortion by fixing the analysis plan and securing acceptance independent of the result, making the published account independent of the outcome.
SupportedVerified against the work's full text
Nosek et al. explain that committing the analysis plan before seeing outcomes keeps test selection and reporting from being shaped by the results.
“With preregistration, prediction is achieved because selection of tests is not influenced by the observed data, and all conducted tests are knowable.”
- The independence condition is exercised through pre-registration, which breaks the distortion by making the account independent of the result.
Mechanisms¶
- Claim Scope Freeze
- Freeze too late and the "before" already reflects the coming shift; press too hard for precision and you record a crisp criterion the speaker never actually held, then unfairly hold them to it — a form of moving-the-goalposts run in reverse.
This sourceExplains that a claim must be frozen before relevant outcomes are observed if prediction is to remain distinguishable from postdiction.
- Freeze too late and the "before" already reflects the coming shift; press too hard for precision and you record a crisp criterion the speaker never actually held, then unfairly hold them to it — a form of moving-the-goalposts run in reverse.
- Convergence Logic Rubric
- Predeclaring the rule is the same discipline that preregistration brings to analysis: fixing the decision rule before the data removes the freedom to interpret results into the answer you wanted.
This sourceDefines preregistration as recording a research plan before observing outcomes to distinguish confirmatory tests from later exploratory choices.
- Predeclaring the rule is the same discipline that preregistration brings to analysis: fixing the decision rule before the data removes the freedom to interpret results into the answer you wanted.
- If-Then Revision Contract
- The discipline that keeps it honest is to fix the trigger and the response before the evidence arrives, tie the trigger to an independently observable measure, and make the rule two-sided so it cannot be invoked only when it suits — the same ex-ante commitment discipline that pre-registration enforces in science.
This sourceDefines preregistration as making analysis commitments before observing outcomes, preventing later opportunistic recasting of predictions.
- The discipline that keeps it honest is to fix the trigger and the response before the evidence arrives, tie the trigger to an independently observable measure, and make the rule two-sided so it cannot be invoked only when it suits — the same ex-ante commitment discipline that pre-registration enforces in science.
- Pre-Registered Benchmark Policy
- Its strength is that it structurally defeats post-hoc benchmark shopping and the garden of forking paths — the quiet manipulation where a comparator chosen after the fact is passed off as principled — by making the timing of the choice auditable.
This sourceDescribes preregistration as a time-stamped plan that makes prespecified analyses distinguishable from post hoc choices and reduces bias from the garden of forking paths.
- Its strength is that it structurally defeats post-hoc benchmark shopping and the garden of forking paths — the quiet manipulation where a comparator chosen after the fact is passed off as principled — by making the timing of the choice auditable.
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 7 of 11 citations: 1 supported, 6 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:ad3c5dcfb535 · see in the full table