Estimating the Reproducibility of Psychological Science.¶
Collaboration, O. S. (2015). Estimating the Reproducibility of Psychological Science. Science, 349(6251).
Cited by¶
16 citations across 14 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Hypothesis Testing (Null vs. Alternative)
- Listed in the references but not attached to a specific claim.
- Primary vs. Secondary Sources
- The same primary/secondary discipline appears in the replication crisis literature: the Open Science Collaboration (2015), in attempting to reproduce 100 published psychology findings, found that secondary reports could only be evaluated by returning to primary materials and protocols, and that fewer than half of the original effects replicated when the primary record was actually re-examined.
Supported in partVerified against the work's full text
Confirms replications of 100 published studies and that only 36% of replications produced significant results, but nowhere states that secondary reports can only be evaluated by returning to primary materials and protocols.
“We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available.”
- The same primary/secondary discipline appears in the replication crisis literature: the Open Science Collaboration (2015), in attempting to reproduce 100 published psychology findings, found that secondary reports could only be evaluated by returning to primary materials and protocols, and that fewer than half of the original effects replicated when the primary record was actually re-examined.
- Priming
- The construct's scientific status varies across these uses — the tight semantic-priming finding is bedrock, the broad behavioral-priming claims require careful evaluation.
Supported in partVerified against the publisher's abstract
The replication project supports the general psychology replication crisis, but the abstract names no social-priming result.
“they find that about one-third to one-half of the original findings were also observed in the replication study.”
- The construct's scientific status varies across these uses — the tight semantic-priming finding is bedrock, the broad behavioral-priming claims require careful evaluation.
- Reproducibility & Replicability
- The replication studies followed the original protocols as closely as possible, typically with larger sample sizes than the original (median replication sample size was 2.5 times the original, attempting to provide >80% power to detect the original effect size)
Supported in partVerified against the work's full text
OSC 2015 confirms protocol fidelity, original materials and high power, but the retrieved passages give no median 2.5x sample size or 80% power figure.
“Nonetheless, collectively these results offer a clear conclusion: A large portion of replications produced weaker evidence for the original findings despite using materials provided by the original authors, review in advance for methodological fidelity, and high statistical power to detect the original effect sizes.”
- Across these metrics, replication rates were approximately: 36% by same-direction significance; 47% by interval overlap; 68% by combined-evidence; 39% by subjective judgment
This sourceOpen Science Collaboration quantified replication rates across multiple success definitions.
Supported in partVerified against the publisher's abstract
The summary supports roughly one-third to one-half replication across criteria, consistent with 36%, 47% and 39%, but gives none of the specific figures, and 68% falls outside it.
“they find that about one-third to one-half of the original findings were also observed in the replication study.”
- The paper's publication in Science generated massive methodological-reform impact: (i) It quantified for the first time the psychology replication problem at scale with rigorous methodology, converting anecdotal concerns into statistical evidence
This sourceOpen Science Collaboration large-scale replication converting anecdotal crisis concerns into statistical evidence.
Supported in partVerified against the publisher's abstract
The paper's own abstract supports a large-scale collaborative replication framed as a first estimate where evidence was limited, but says nothing about reform impact or priority.
“We conducted a large-scale, collaborative effort to obtain an initial estimate of the reproducibility of psychological science.”
- The replication studies followed the original protocols as closely as possible, typically with larger sample sizes than the original (median replication sample size was 2.5 times the original, attempting to provide >80% power to detect the original effect size)
- Statistical Inference
- Statistical Significance (p-Value)
- Technical Debt
- The pattern ports further. The deliberate-versus-inadvertent analysis into organisational design: the distinction between debt taken knowingly and debt that silently became permanent transfers from software into change management, where the first task is again to discover which workarounds have hardened into infrastructure. Refactoring discipline into scientific replication: the practice of regularly revisiting and consolidating prior work transfers into replication audits and meta-analyses understood as "literature refactoring," paying down the replication debt accumulated when claims were accepted ungrounded.
This sourceReplications of 100 studies reproduced only ~36% of originally significant effects, with effect sizes halved — quantifying the replication debt accumulated from ungrounded claims that audits and meta-analyses must pay down.
Supported in partVerified against the work's full text
Supplies a replication audit showing that many published claims fail to replicate; it does not state that refactoring discipline 'transfers' or name a 'replication debt'.
“Scientific claims should not gain credence because of the status or authority of their originator but by the replicability of their supporting evidence.”
- The pattern ports further. The deliberate-versus-inadvertent analysis into organisational design: the distinction between debt taken knowingly and debt that silently became permanent transfers from software into change management, where the first task is again to discover which workarounds have hardened into infrastructure. Refactoring discipline into scientific replication: the practice of regularly revisiting and consolidating prior work transfers into replication audits and meta-analyses understood as "literature refactoring," paying down the replication debt accumulated when claims were accepted ungrounded.
- Texas Sharpshooter Fallacy
- In empirical science, HARKing (hypothesizing after results are known), p-hacking, and the garden of forking paths are all the same structural failure operating at the scale of a literature, with the replication crisis as the aggregate symptom.
This sourceLarge-scale replication project documenting the replication crisis as an aggregate symptom of practices including p-hacking and HARKing.
Supported in partVerified against the publisher's abstract
The summary documents low replication in psychology, but does not link it to HARKing, p-hacking or forking paths.
“they find that about one-third to one-half of the original findings were also observed in the replication study.”
- In empirical science, HARKing (hypothesizing after results are known), p-hacking, and the garden of forking paths are all the same structural failure operating at the scale of a literature, with the replication crisis as the aggregate symptom.
- Transparency
- Verification
Domain-specific¶
- HARKing (Hypothesizing After the Results are Known)
- Type S Error
- Sign flips between high-profile original studies and their high-powered replications — multiple reversal cases in behavioral science — are the downstream observable signature of Type S errors materializing across a literature
This sourceReports high-powered replications of 100 psychology studies in which 17 replication effects had the opposite sign to the original.
Supported in partVerified against the source
- Sign flips between high-profile original studies and their high-powered replications — multiple reversal cases in behavioral science — are the downstream observable signature of Type S errors materializing across a literature
Mechanisms¶
- Independent Replication
- If it shrinks to nothing, the "settled" finding was, in the language of the wider reckoning over reproducibility, an artifact of the conditions that produced it rather than a fact about students.
This sourceDocuments widespread attenuation of effects in psychology replications while warning that a single nonreplication does not by itself establish that the original finding was false.
- If it shrinks to nothing, the "settled" finding was, in the language of the wider reckoning over reproducibility, an artifact of the conditions that produced it rather than a fact about students.
- Replication Study
- Either outcome is a data point about how much the finding can be trusted outside the room it was born in — the kind of check that the broader replication crisis has shown is indispensable.
This sourceTreats both positive and negative replication outcomes as cumulative evidence about reproducibility, illustrating why replication is a defining scientific check.
- Either outcome is a data point about how much the finding can be trusted outside the room it was born in — the kind of check that the broader replication crisis has shown is indispensable.
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 9 of 16 citations: 9 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Was it audited? Yes. A second, independent pass read the citation against the article text and recorded a verdict.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:b59904983efe · see in the full table