False-Positive Psychology¶
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359-1366.
Cited by¶
11 citations across 11 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Motivated Reasoning
- A pre-registered stopping point, or a sequential rule with its thresholds set in advance, would have produced a different verdict from identical data.
This sourceQuantifies how sampling until a threshold is crossed invalidates a fixed-n significance threshold and inflates the apparent strength of evidence.
Supported in partVerified against the publisher's abstract
The abstract states that undisclosed flexibility in data collection sharply inflates false-positive rates, which supports the claim's frame but not its counterfactual that pre-registration would have changed this verdict.
“flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates”
- A pre-registered stopping point, or a sequential rule with its thresholds set in advance, would have produced a different verdict from identical data.
- Reproducibility & Replicability
- The selection-bias diagnostic suggests that much of the observed effect reflects customer-quality selection rather than causal feature effect
This sourceSimmons false-positive-psychology framework documenting p-hacking and garden-of-forking-paths flexibility.
- The selection-bias diagnostic suggests that much of the observed effect reflects customer-quality selection rather than causal feature effect
- Researcher Degrees of Freedom
- In statistics and empirical science — the canonical home — ordinary, plausible analytic flexibility can raise the false-positive rate of a nominal five-percent test above sixty percent,
This sourceCoins "researcher degrees of freedom" and shows ordinary analytic flexibility can push the false-positive rate above 60%.
Supported in partVerified against the publisher's abstract
The abstract confirms that researcher flexibility sharply inflates the false-positive rate of a nominal five-percent test, but it states no specific sixty-percent figure.
“we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not.”
- In statistics and empirical science — the canonical home — ordinary, plausible analytic flexibility can raise the false-positive rate of a nominal five-percent test above sixty percent,
- Termination Condition
- It also exposes the dual error of premature termination: a stopping rule chosen to make the result come out the right way, as in early stopping in clinical trials, optional-stopping p-hacking, or ending negotiations when you are winning.
This sourceDemonstrates that optional stopping (data-dependent choice of when to stop) inflates false-positive rates.
Supported in partVerified against the publisher's abstract
The abstract backs the frame — undisclosed flexibility inflates false positives — but does not itself name optional stopping or a stopping rule.
“flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates”
- It also exposes the dual error of premature termination: a stopping rule chosen to make the result come out the right way, as in early stopping in clinical trials, optional-stopping p-hacking, or ending negotiations when you are winning.
Domain-specific¶
- Implicit Egotism
- Base rates, birth cohorts, demographic clustering, repeated exposure, and researcher degrees of freedom can manufacture those larger correlations
Supported in partVerified against the source
- Base rates, birth cohorts, demographic clustering, repeated exposure, and researcher degrees of freedom can manufacture those larger correlations
Mechanisms¶
- Anticipatory Offset Dashboard
- The discipline is to pre-register the indicator set, baselines, and watchlist before announcement, so the display cannot be quietly re-cut to flatter the result
This sourceShows that undisclosed flexibility in choosing variables, conditions, and analyses permits selectively favorable results, and uses advance decisions and full disclosure to constrain it.
- The discipline is to pre-register the indicator set, baselines, and watchlist before announcement, so the display cannot be quietly re-cut to flatter the result
- Assumption Audit Checklist
- By forcing enumeration before selection, it converts researcher degrees of freedom — the many undocumented choices between data and result — from an invisible risk into a governed list.
This sourceIdentifies researcher degrees of freedom as consequential, often undisclosed choices and proposes explicit decision and disclosure requirements to govern them.
- By forcing enumeration before selection, it converts researcher degrees of freedom — the many undocumented choices between data and result — from an invisible risk into a governed list.
- Multiverse Analysis Report
- Each combination is a legitimate analysis, and reporting only the one that yields a crisp, significant effect would be researcher degrees of freedom at work.
This sourceShows that undisclosed flexibility in data collection and analysis can be used to select and report a statistically significant result.
- Each combination is a legitimate analysis, and reporting only the one that yields a crisp, significant effect would be researcher degrees of freedom at work.
- Protocol Deviation Register
- Its strength is that it converts messy, inevitable real-world exceptions into governed, auditable data, and it blocks outcome-dependent cleaning by fixing the disposition rules before anyone sees which way they cut.
This sourceShows that outcome-dependent flexibility in data exclusion and analysis inflates false positives and recommends committing to data-collection and analysis rules before observing the results.
- Its strength is that it converts messy, inevitable real-world exceptions into governed, auditable data, and it blocks outcome-dependent cleaning by fixing the disposition rules before anyone sees which way they cut.
- Research Stopping Rule
- Its signature hazard is optional stopping — halting collection or analysis at the moment the result looks best, which inflates false findings and is a live methodological problem wherever data accumulate over time.
This sourceDemonstrates that undisclosed flexibility in deciding when to stop collecting data inflates false-positive rates in studies where observations accumulate sequentially.
- Its signature hazard is optional stopping — halting collection or analysis at the moment the result looks best, which inflates false findings and is a live methodological problem wherever data accumulate over time.
- Selection-Differential Cohort Analysis
- Its classic misuse is running the analysis to confirm a selection story already believed, quietly adjusting the model until the expected differential appears.
This sourcePsychological Science. 22(11): 1359–1366. (2011). Simmons, Nelson, and Simonsohn show how motivated use of undisclosed analytical flexibility can make a desired result appear significant.
- Its classic misuse is running the analysis to confirm a selection story already believed, quietly adjusting the model until the expected differential appears.
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 4 of 11 citations: 4 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Was it audited? Yes. A second, independent pass read the citation against the article text and recorded a verdict.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:eb9a3dff0932 · see in the full table