False-Positive Psychology¶
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359-1366.
Cited by¶
6 citations across 6 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Motivated Reasoning
- A pre-registered stopping point, or a sequential rule with its thresholds set in advance, would have produced a different verdict from identical data.
This sourceQuantifies how sampling until a threshold is crossed invalidates a fixed-n significance threshold and inflates the apparent strength of evidence.
- A pre-registered stopping point, or a sequential rule with its thresholds set in advance, would have produced a different verdict from identical data.
- Reproducibility & Replicability
- The selection-bias diagnostic suggests that much of the observed effect reflects customer-quality selection rather than causal feature effect
This sourceSimmons false-positive-psychology framework documenting p-hacking and garden-of-forking-paths flexibility.
- The selection-bias diagnostic suggests that much of the observed effect reflects customer-quality selection rather than causal feature effect
- Researcher Degrees of Freedom
- In statistics and empirical science — the canonical home — ordinary, plausible analytic flexibility can raise the false-positive rate of a nominal five-percent test above sixty percent,
This sourceCoins "researcher degrees of freedom" and shows ordinary analytic flexibility can push the false-positive rate above 60%.
- In statistics and empirical science — the canonical home — ordinary, plausible analytic flexibility can raise the false-positive rate of a nominal five-percent test above sixty percent,
- Termination Condition
- It also exposes the dual error of premature termination: a stopping rule chosen to make the result come out the right way, as in early stopping in clinical trials, optional-stopping p-hacking, or ending negotiations when you are winning.
This sourceDemonstrates that optional stopping (data-dependent choice of when to stop) inflates false-positive rates.
- It also exposes the dual error of premature termination: a stopping rule chosen to make the result come out the right way, as in early stopping in clinical trials, optional-stopping p-hacking, or ending negotiations when you are winning.
Domain-specific¶
- Implicit Egotism
- Base rates, birth cohorts, demographic clustering, repeated exposure, and researcher degrees of freedom can manufacture those larger correlations
This sourceThe source of the researcher-degrees-of-freedom clause - a general demonstration that undisclosed flexibility in data collection, analysis and reporting is enough on its own to produce significant support for a false hypothesis; the demographic and cohort confounds listed alongside it are Simonsohn (2011)'s.
- Base rates, birth cohorts, demographic clustering, repeated exposure, and researcher degrees of freedom can manufacture those larger correlations
Mechanisms¶
- Anticipatory Offset Dashboard
- The discipline is to pre-register the indicator set, baselines, and watchlist before announcement, so the display cannot be quietly re-cut to flatter the result
This sourceShows that undisclosed flexibility in choosing variables, conditions, and analyses permits selectively favorable results, and uses advance decisions and full disclosure to constrain it.
- The discipline is to pre-register the indicator set, baselines, and watchlist before announcement, so the display cannot be quietly re-cut to flatter the result
Verification¶
This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.
Registry ID ref:eb9a3dff0932 · see in the full table