The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No 'Fishing Expedition' or 'p-Hacking' and the Research Hypothesis Was Posited Ahead of Time¶
Gelman, A., & Loken, E. (2013). The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No 'Fishing Expedition' or 'p-Hacking' and the Research Hypothesis Was Posited Ahead of Time: why multiple comparisons and p-hacking likely afflict us all. Unpublished manuscript.
Cited by¶
8 citations across 8 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Multiple Comparisons Correction
- Modern reporting norms — CONSORT (trials), STROBE (observational), pre-registration (psychology) — mandate the distinction between primary confirmatory tests (pre-specified, few in number, deserving of single-test α) and secondary or exploratory tests (post-hoc, numerous, requiring correction or explicit hypothesis-generating flagging)
This sourceGelman Loken garden forking paths multiplicity selection bias p-hacking.
Supported in partVerified against the work's full text
Establishes the forking-paths idea that unchosen analysis paths matter for p-values, but stresses data-contingent choices without fishing, not conducted-but-unreported tests, and treats its framework as frequentist.
“But if we average over all possible data that could have occurred, we need to look at the entire garden of forking paths”
- Modern reporting norms — CONSORT (trials), STROBE (observational), pre-registration (psychology) — mandate the distinction between primary confirmatory tests (pre-specified, few in number, deserving of single-test α) and secondary or exploratory tests (post-hoc, numerous, requiring correction or explicit hypothesis-generating flagging)
- Researcher Degrees of Freedom
- and the "garden of forking paths" extends the point even to single-comparison analyses that are data-dependent.
This sourceExtends the multiplicity problem to single, data-dependent analyses.
SupportedVerified against the work's full text
Gelman and Loken's abstract states that data-contingent analysis choices create a multiple comparisons problem even when only a single analysis is performed.
“Researcher degrees of freedom can lead to a multiple comparisons problem, even in settings where researchers perform only a single analysis on their data.”
- and the "garden of forking paths" extends the point even to single-comparison analyses that are data-dependent.
Mechanisms¶
- Blinded Data Analysis Plan
- Its strength is closing off the garden of forking paths — the many defensible-looking analytic choices whose selection, made after seeing the answer, silently manufactures significance.
This sourceShows how data-contingent choices among reasonable analyses create hidden multiplicity and make statistical significance too easy to obtain.
- Its strength is closing off the garden of forking paths — the many defensible-looking analytic choices whose selection, made after seeing the answer, silently manufactures significance.
- Interaction Term Gate
- Its central failure mode is theater: a gate that demands a "rationale" but accepts any plausible-sounding story admits nearly everything, and one that ignores how many interactions were tried lets multiplicity smuggle false positives through one at a time.
This sourceShows that data-dependent analytic choices create an uncounted multiple-comparisons problem that can yield apparently significant results even without deliberate p-hacking.
- Its central failure mode is theater: a gate that demands a "rationale" but accepts any plausible-sounding story admits nearly everything, and one that ignores how many interactions were tried lets multiplicity smuggle false positives through one at a time.
- Micro-Experiment Sequence
- Its failure mode is subtle: an undisciplined branching sequence with a rich trace is a garden of forking paths — with enough experiments and enough freedom to reinterpret after the fact, you will eventually "find" a result that is noise dressed as signal.
This sourceShows how data-contingent analytical choices can make pure noise appear statistically significant.
- Its failure mode is subtle: an undisciplined branching sequence with a rich trace is a garden of forking paths — with enough experiments and enough freedom to reinterpret after the fact, you will eventually "find" a result that is noise dressed as signal.
- Model-Revision Experiment Log
- And iteratively chasing residual patterns is itself a garden of forking paths: with enough attempts, noise will masquerade as structure, so a log without held-out verdicts launders overfitting into a paper trail.
This sourceExplains how many data-dependent analytic paths can make noise appear to be a meaningful pattern.
- And iteratively chasing residual patterns is itself a garden of forking paths: with enough attempts, noise will masquerade as structure, so a log without held-out verdicts launders overfitting into a paper trail.
- Prespecified Adjusted Estimation Plan
- And a plan that is prespecified in name but loose in substance still leaves room for the garden of forking paths — the many defensible analytic choices that, taken after seeing the data, quietly bias the result.
This sourceExplains how many reasonable, data-contingent analytic choices can bias results even when the research hypothesis was specified in advance.
- And a plan that is prespecified in name but loose in substance still leaves room for the garden of forking paths — the many defensible analytic choices that, taken after seeing the data, quietly bias the result.
- Residual Root-Cause Review
- Its weakness is that a review is exactly the setting in which a group fits a compelling just-so story to what is really noise — given enough analytic flexibility, some explanation always fits
This sourceShows how data-dependent analytic choices create a garden of forking paths in which noisy data can yield apparently compelling findings without deliberate fishing.
- Its weakness is that a review is exactly the setting in which a group fits a compelling just-so story to what is really noise — given enough analytic flexibility, some explanation always fits
Verification¶
Does it exist? Not checked yet. This entry carries no identifier to resolve. It was extracted from the citation as written in the article, normalized, and deduplicated against the rest of the registry.
Does it back the claim? Read against the text for 2 of 8 citations: 1 supported, 1 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:894136fbaf58 · see in the full table