The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No 'Fishing Expedition' or 'p-Hacking' and the Research Hypothesis Was Posited Ahead of Time¶
Gelman, A., & Loken, E. (2013). The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No 'Fishing Expedition' or 'p-Hacking' and the Research Hypothesis Was Posited Ahead of Time: why multiple comparisons and p-hacking likely afflict us all. Unpublished manuscript.
Cited by¶
3 citations across 3 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Multiple Comparisons Correction
- Modern reporting norms — CONSORT (trials), STROBE (observational), pre-registration (psychology) — mandate the distinction between primary confirmatory tests (pre-specified, few in number, deserving of single-test α) and secondary or exploratory tests (post-hoc, numerous, requiring correction or explicit hypothesis-generating flagging)
This sourceGelman Loken garden forking paths multiplicity selection bias p-hacking.
- Modern reporting norms — CONSORT (trials), STROBE (observational), pre-registration (psychology) — mandate the distinction between primary confirmatory tests (pre-specified, few in number, deserving of single-test α) and secondary or exploratory tests (post-hoc, numerous, requiring correction or explicit hypothesis-generating flagging)
- Researcher Degrees of Freedom
- and the "garden of forking paths" extends the point even to single-comparison analyses that are data-dependent.
This sourceExtends the multiplicity problem to single, data-dependent analyses.
- and the "garden of forking paths" extends the point even to single-comparison analyses that are data-dependent.
Mechanisms¶
- Micro-Experiment Sequence
- Its failure mode is subtle: an undisciplined branching sequence with a rich trace is a garden of forking paths — with enough experiments and enough freedom to reinterpret after the fact, you will eventually "find" a result that is noise dressed as signal.
This sourceShows how data-contingent analytical choices can make pure noise appear statistically significant.
- Its failure mode is subtle: an undisciplined branching sequence with a rich trace is a garden of forking paths — with enough experiments and enough freedom to reinterpret after the fact, you will eventually "find" a result that is noise dressed as signal.
Verification¶
This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.
Registry ID ref:894136fbaf58 · see in the full table