Estimating the Reproducibility of Psychological Science.

Collaboration, O. S., & GM, S. (2015). Estimating the Reproducibility of Psychological Science. Science, 349(6251).

Type

Journal article

Intellectual base

Primary research

Year

2015

DOI

10.1126/science.aac4716

Link

https://doi.org/10.1126/science.aac4716

Cited from

computer_science, engineering_design, history_historiography, political_science, rhetoric, statistics_experimental_design

Cited by

13 citations across 11 artifacts.

Each citation links to the sentence it supports in the citing article.

Primes

• Hypothesis Testing (Null vs. Alternative)
  • Listed in the references but not attached to a specific claim.
• Primary vs. Secondary Sources
  • The same primary/secondary discipline appears in the replication crisis literature: the Open Science Collaboration (2015), in attempting to reproduce 100 published psychology findings, found that secondary reports could only be evaluated by returning to primary materials and protocols, and that fewer than half of the original effects replicated when the primary record was actually re-examined.
• Priming
  • The construct's scientific status varies across these uses — the tight semantic-priming finding is bedrock, the broad behavioral-priming claims require careful evaluation.
• Reproducibility & Replicability
  • The replication studies followed the original protocols as closely as possible, typically with larger sample sizes than the original (median replication sample size was 2.5 times the original, attempting to provide >80% power to detect the original effect size)
  • Across these metrics, replication rates were approximately: 36% by same-direction significance; 47% by interval overlap; 68% by combined-evidence; 39% by subjective judgment

    This sourceOpen Science Collaboration quantified replication rates across multiple success definitions.

  • The paper's publication in Science generated massive methodological-reform impact: (i) It quantified for the first time the psychology replication problem at scale with rigorous methodology, converting anecdotal concerns into statistical evidence

    This sourceOpen Science Collaboration large-scale replication converting anecdotal crisis concerns into statistical evidence.

• Statistical Inference
  • Pre-registration (specifying hypotheses, methods, and analysis before seeing data) and open science practices reduce these problems by making it difficult to bend the analysis post hoc.
• Statistical Significance (p-Value)
  • The replication crisis beginning around 2011 (Bem ESP paper; Open Science Collaboration 2015 large-scale replication study reporting ~36% replication rate) prompted widespread methodological reform
• Technical Debt
  • The pattern ports further. The deliberate-versus-inadvertent analysis into organisational design: the distinction between debt taken knowingly and debt that silently became permanent transfers from software into change management, where the first task is again to discover which workarounds have hardened into infrastructure. Refactoring discipline into scientific replication: the practice of regularly revisiting and consolidating prior work transfers into replication audits and meta-analyses understood as "literature refactoring," paying down the replication debt accumulated when claims were accepted ungrounded.

    This sourceReplications of 100 studies reproduced only ~36% of originally significant effects, with effect sizes halved — quantifying the replication debt accumulated from ungrounded claims that audits and meta-analyses must pay down.

• Texas Sharpshooter Fallacy
  • In empirical science, HARKing (hypothesizing after results are known), p-hacking, and the garden of forking paths are all the same structural failure operating at the scale of a literature, with the replication crisis as the aggregate symptom.

    This sourceLarge-scale replication project documenting the replication crisis as an aggregate symptom of practices including p-hacking and HARKing.

• Transparency
  • T2: Disclosure quantity versus accessibility and usefulness.
• Verification
  • (Under R17a in the project-06 hierarchy, `reproducibility_replicability` re-parents from `validation` to `verification` for exactly this reason.)

Mechanisms

• Replication Study
  • Either outcome is a data point about how much the finding can be trusted outside the room it was born in — the kind of check that the broader replication crisis has shown is indispensable.

    This sourceTreats both positive and negative replication outcomes as cumulative evidence about reproducibility, illustrating why replication is a defining scientific check.

Verification

This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.

Registry ID ref:b59904983efe · see in the full table