Skip to content

Researcher Degrees of Freedom

Prime #
1139
Origin domain
Statistics Probability Research Reliability
Subdomain
research validity → Statistics Probability Research Reliability

Core Idea

Researcher degrees of freedom name the unpinned analytic choices that sit between a research question and a reported result — which subjects to exclude, which transformations to apply, which covariates to include, which test to run, when to stop collecting data, which subgroup to report, which outcome to feature. Each choice is locally defensible; the structural problem is that the garden of forking paths explored silently in private becomes a single declared comparison in public. This silent multiplicity inflates the false-positive rate by orders of magnitude even when no individual decision was made in bad faith.

The pattern is not bias, fraud, or motivated reasoning in any single-choice sense. It is the gap between a flexible decision tree and the singular report, where the flexibility itself is the source of inferential failure. The structure has a definite shape: a research question or estimation target; an analytic decision tree with a branch at every unfixed choice; a silent comparison budget, equal to the size of the tree actually explored; a visible single report, one leaf summarised as the result; an inferential warrant gap between the declared and the exercised multiplicity; and a pre-commitment lever — registration, holdout, multiverse — that can collapse or reveal the budget.

What makes the pattern distinctive is that the multiplicity is invisible. Declared multiple testing has standard corrections; here the comparisons are made silently, as analytic choices rather than explicit tests, so the corrections do not apply because no one can see the budget to correct for it. The inferential warrant depends not on the comparison reported but on the size of the tree from which it was selected — a counterfactual ("which fraction of the tree would have been reported had it come out positive?") that no reader can audit.

How would you explain it like I'm…

 

No faithful explanation at this level. Two of three generators marked this na: any 5-year-old framing collapses the concept into 'the scientist cheated or lied,' but the defining point is that each individual choice is honest and locally reasonable — the inferential failure comes only from the unseen many-paths (garden-of-forking-paths) structure, which the cheating frame erases.

Secret Forking Paths

Researcher degrees of freedom are all the small, reasonable choices a scientist makes between asking a question and reporting an answer — like which data to leave out, which math to use, or when to stop collecting. Each choice seems fine on its own, and nobody is cheating. The trouble is that the scientist quietly tried many paths in private but reports only the one that worked, as if it were the only path tried. Imagine taking twenty different routes through a maze but telling people you took just one. That hidden 'I tried many ways' makes a lucky result look much more certain than it really is.

Garden Of Forking Paths

Researcher degrees of freedom are the unpinned analytic choices that sit between a research question and a reported result — which subjects to exclude, which transformation to apply, which covariates to include, which test to run, when to stop collecting data, which subgroup to report. Each choice is locally defensible, and crucially none of them is fraud or bias in any single-choice sense. The structural problem is the gap: a 'garden of forking paths' explored silently in private becomes a single declared comparison in public. That hidden multiplicity inflates the false-positive rate by orders of magnitude even when every individual decision was made in good faith. Unlike openly running many tests — which has standard corrections — here the comparisons are invisible, so no one can see the budget to correct for it.

 

Researcher degrees of freedom name the unpinned analytic choices that sit between a research question and a reported result — which subjects to exclude, which transformations to apply, which covariates to include, which test to run, when to stop collecting data, which subgroup to report, which outcome to feature. Each choice is locally defensible; the structural problem is that the garden of forking paths explored silently in private becomes a single declared comparison in public, and this silent multiplicity inflates the false-positive rate by orders of magnitude even with no individual decision made in bad faith. The pattern is not bias, fraud, or motivated reasoning in any single-choice sense; it is the gap between a flexible decision tree and the singular report, where the flexibility itself is the source of inferential failure. The structure has a definite shape: a question or estimation target; an analytic decision tree branching at every unfixed choice; a silent comparison budget equal to the size of the tree actually explored; a visible single report (one leaf summarized as 'the result'); an inferential warrant gap between declared and exercised multiplicity; and a pre-commitment lever — registration, holdout, multiverse — that can collapse or reveal the budget. What makes it distinctive is that the multiplicity is invisible: declared multiple testing has standard corrections, but here the comparisons are made silently as analytic choices rather than explicit tests, so the corrections don't apply because no one can see the budget. The warrant depends not on the comparison reported but on the size of the tree it was selected from — a counterfactual no reader can audit.

Structural Signature

the question or estimation target — the analytic decision tree branching at every unfixed choice — the silent comparison budget equal to the tree actually explored — the visible single report, one leaf declared as the result — the warrant gap between declared and exercised multiplicity — the pre-commitment lever that can collapse or reveal the budget

A process exhibits this pattern when each of the following holds:

  • A question or estimation target. A research, evaluation, or inference goal whose answer will be reported.
  • An analytic decision tree. A set of unpinned choices — exclusions, transformations, covariates, tests, stopping rules, subgroups, outcomes — each branching the path from question to result; every branch is individually defensible.
  • A silent comparison budget. The de-facto multiplicity is the size of the tree actually explored, exercised privately as analytic choices rather than as declared tests, so standard multiple-comparison corrections do not see it.
  • A visible single report. One leaf of the tree is summarised publicly as the result, as though it were a single pre-planned comparison.
  • A warrant gap. The inferential warrant depends not on the comparison reported but on the size of the tree from which it was selected — a counterfactual no reader can audit.
  • A pre-commitment lever. Registration, holdout separation, or multiverse reporting can collapse the silent multiplicity into something visible or fix the tree before the data are seen.

These compose so that the pathology requires no bad faith in any single choice: invisible multiplicity inflates false confidence, and every remedy is the same move — make the comparison budget visible or pre-committed.

What It Is Not

  • Not regret. regret is a backward-looking valuation of a forgone alternative outcome; researcher degrees of freedom is a forward inferential pathology of silent multiplicity inflating false confidence. One is about counterfactual loss felt; the other about an un-auditable comparison budget.
  • Not selection bias. selection_bias distorts a sample at data collection; researcher degrees of freedom distorts inference at analysis, through choices made after the data are in hand. Sampling vs. analytic flexibility.
  • Not multiple comparisons. multiple_comparisons_correction handles declared tests with known corrections; the whole pathology here is that the comparisons are silent — analytic choices no one can see, so corrections cannot be applied. (Same statistical root, opposite visibility.)
  • Not overfitting. overfitting is a model fitting noise in training data; researcher degrees of freedom is a reporting pathology where one leaf of an analytic tree is declared as if pre-planned. Backtest overfitting is one instance, not the prime.
  • Not confounding. confounding is a causal-structure error — a lurking common cause; researcher degrees of freedom is an inferential-warrant error from selecting a result out of a large tree, even with no causal confound.
  • Common misclassification. Hunting for the one biased choice and auditing each decision (finding all defensible) while the warrant is already gone. Catch it by shifting from "was any choice biased?" to "how many de-facto comparisons did this report collapse into one?"

Broad Use

The pattern recurs wherever flexible analysis culminates in a singular declared result. In statistics and empirical science — the canonical home — ordinary, plausible analytic flexibility can raise the false-positive rate of a nominal five-percent test above sixty percent, and the "garden of forking paths" extends the point even to single-comparison analyses that are data-dependent. In machine-learning evaluation, tuning on the test set, model selection across many architectures, choice of benchmark suite, and prompt selection each reproduce the structure: many silent comparisons summarised as one. In financial backtesting, strategy researchers explore hundreds of variants on the same historical data and report the best, the pattern known there as backtest overfitting. In policy and program evaluation, the choice of outcome window, comparison group, control variables, and treatment definition each introduces flexibility that survives into the published estimate. In audit and accounting, the choice of inventory method, depreciation schedule, accrual timing, and segment definition gives the same forking structure for one underlying business. In investigative journalism and intelligence analysis, the choice of framing, which sources to weight, and which timeframes to compare reproduces the identical structure under a different label.

Clarity

The label clarifies a phenomenon that single-study failure modes cannot explain: the systematic over-statement of confidence across an entire field, even when each study looks careful. Once one sees the flexibility as a multiplicative comparison budget rather than as care, the inferential pathology snaps into focus — the field is not unlucky or sloppy study by study; it is selecting from a large tree and reporting the selection as though it were the whole.

The frame also forces a separation between two questions that are routinely fused: whether each individual analytic choice was defensible (usually yes), and whether the set of choices, taken as a silent comparison budget, warrants the reported confidence (usually no). By making the budget the object of attention rather than any single decision, the frame defuses the unproductive search for the one bad choice and replaces it with a structural question about the size of the tree. The clarifying move is to stop asking "was this analysis biased?" and start asking "how many de-facto comparisons did this report quietly collapse into one?"

Manages Complexity

The pattern compresses a sprawling list of micro-choices — exclusions, transformations, covariates, tests, stopping rules, subgroups, outcomes — into a single structural quantity: how many de-facto comparisons did the report quietly collapse? An analyst can communicate the entire diagnosis in one phrase, and the remedy menu follows directly from it, because every remedy is a way of either pre-committing the tree or revealing it. A confusing audit of dozens of individual decisions becomes one estimate of the comparison budget.

The compression matters because it reframes the analysis itself as a decision tree of branches with implicit selection, where the visible report is one leaf and the inferential warrant depends on the size of the tree. This lets the analyst reason about inferential confidence as a function of the comparison budget exercised rather than the comparison reported — a single lever — and it makes the remedies commensurable, since pre-registration, holdout separation, and multiverse reporting are all the same structural move (collapse the silent multiplicity into something visible or pre-committed) expressed in different vocabularies.

Abstract Reasoning

The frame licenses a specific inference: an analysis should be modelled as a tree whose branches are the unfixed choices, with the reported result one leaf, and the warrant of that result discounted by the size of the tree that could have been explored and selectively reported. From this follows the central deduction — that confidence stated as though the report were a single pre-planned comparison is unwarranted whenever the underlying tree was large and data-dependent, regardless of the good faith of any branch.

The reasoning carries a heavy institutional and normative load that the frame does not hide. The very category "researcher" presupposes a scientific or evaluative reporting institution; the framing of false-positive inflation as an inferential failure presupposes norms about what a reported result is supposed to warrant. The pattern is therefore strongly framed rather than structural: its vocabulary travels only with translation, it carries an explicit evaluative charge, and it is bound to human practices of analysis and reporting. The structural core that could be extracted — silent multiplicity collapsed into a singular declaration — is portable, but as the prime is written it imports the reporting-norm context wholesale, which is what makes the inference about warrant meaningful rather than merely descriptive.

Knowledge Transfer

The remedies travel together with the diagnosis because the roles map across domains: the decision tree maps to analytic-choice trees, hyperparameter and architecture sweeps, backtest variant spaces, or specification spaces; the silent comparison budget maps to the size of any of these; the visible single report maps to the published effect, the headline benchmark number, the reported strategy return, or the published estimate; and the pre-commitment lever maps to pre-registration, held-out test sets, or multiverse and specification-curve reporting. Because the roles correspond, each remedy is the same structural move — collapse silent multiplicity into something visible or pre-committed — even though every domain invented its own name for it.

The documented transfers are concrete. Pre-registration fixes the analytic plan before the data are seen so flexibility cannot be exercised silently; multiverse analysis reports results across many plausible branches and lets the reader see the distribution; held-out test sets physically separate exploration data from evaluation data so exploration multiplicity cannot leak into the evaluation; reporting standards make the comparison budget visible by requiring declaration of the branches considered; specification curves visualise the distribution of estimates across all defensible specifications; and adversarial collaboration lets opposing teams set the analytic plan together, eliminating one-sided flexibility. The same diagnosis — a silent decision tree exceeding the declared comparisons — and the same intervention menu — pre-commitment, holdout separation, multiverse reporting — work in finance backtests, machine-learning benchmarks, policy evaluations, and audits, which is the cross-domain signature: the pattern travels with its mechanism and its remedy set, not merely its vocabulary. The transfer remains framed rather than purely structural, because in every destination domain it carries the same normative claim that a singular declared result must warrant the confidence placed in it, and that claim presupposes a reporting institution whose standards the silent multiplicity violates.

Examples

Formal/abstract

Consider a psychology experiment testing whether a priming manipulation affects a behavioural outcome, the canonical home of the prime. The question is fixed, but the path from data to result branches at every unpinned choice. There are two candidate outcome measures (reaction time, accuracy); three defensible exclusion rules (drop responses faster than 200 ms, slower than 3 s, or beyond 2.5 SD); a choice of whether to log-transform skewed reaction times; the option to control for age and sex or not; and two plausible subgroups (the effect "in women," "in the high-anxiety subset"). These choices multiply: even this modest tree has roughly \(2 \times 3 \times 2 \times 2 \times 3 = 72\) analytic paths. If each path is an independent-ish test at a nominal \(\alpha = 0.05\), the probability that at least one yields \(p < 0.05\) under a true null is \(1 - 0.95^{72} \approx 0.97\) — near-certain. This is the silent comparison budget: the researcher explores the tree privately, finds the one leaf where "the priming effect is significant in women after excluding fast responders and controlling for age," and writes it up as a single declared comparison. No individual choice was made in bad faith; each is defensible. But the warrant gap is total — the reader sees one \(p\)-value and cannot audit the 72-path tree it was selected from, so the nominal 5% false-positive rate is a fiction. The pre-commitment lever collapses the budget: a pre-registered analysis plan fixes outcome, exclusions, transform, and covariates before the data are seen, so the tree shrinks to one path and the reported \(p\) means what it claims; a multiverse analysis instead reports the effect across all 72 specifications, letting the reader see the distribution rather than the cherry.

Mapped back: the priming hypothesis is the question, the outcome/exclusion/transform/covariate/subgroup choices are the analytic decision tree, the 72 paths are the silent comparison budget, the single written-up significant result is the visible report, the un-auditable gap between nominal and effective \(\alpha\) is the warrant gap, and pre-registration or multiverse reporting is the pre-commitment lever.

Applied/industry

Quantitative financial backtesting reproduces the identical structure under the name "backtest overfitting," and the substrate is the same fixed historical price series. A research team wants a profitable trading strategy. The decision tree branches over lookback window (20, 50, 200 days), entry threshold, which technical indicators to combine, the universe of assets, the rebalancing frequency, and the in-sample date range — easily thousands of variant strategies, all run against the same market history. The silent comparison budget is the full set of variants tested in private; the visible single report is the one strategy with the best Sharpe ratio, presented to the investment committee as "our momentum strategy returns 18% annually with a Sharpe of 2.1." The warrant gap is exactly the prime's: a Sharpe of 2.1 selected as the maximum over 2,000 variants is statistically unremarkable — you expect a high-Sharpe outlier by chance alone — but the committee sees one number and cannot discount it for the size of the tree it won. The remedies are the same structural move as pre-registration, in finance vocabulary: hold out a test period the researchers never touch during exploration (physically separating exploration data from evaluation data so exploration multiplicity cannot leak in); report the deflated Sharpe ratio that corrects for the number of trials; and run the strategy forward on genuinely new data before committing capital. The same diagnosis and remedy menu govern machine-learning benchmark gaming (tuning architectures and prompts on the test set, then reporting the best as a single headline number) and policy-evaluation specification search.

Mapped back: the profitable-strategy goal is the question, the lookback/threshold/indicator/universe choices are the decision tree, the thousands of tested variants are the silent comparison budget, the best-Sharpe strategy is the visible report, the un-discounted selection-from-the-max is the warrant gap, and holdout periods plus deflated-Sharpe correction are the pre-commitment levers — the same framed structure operating across empirical science, quantitative finance, and ML evaluation.

Structural Tensions

T1 — Individual Choice Defensibility versus Set-Level Warrant (scalar). The pattern requires no bad faith in any single branch — each exclusion, transform, and covariate is locally defensible — yet the set, taken as a silent comparison budget, destroys the reported confidence. The failure mode is the unproductive hunt for the one bad choice, auditing each decision and finding all defensible while the warrant is gone. Diagnostic: stop asking "was this analysis biased?" and ask "how many de-facto comparisons did this report quietly collapse into one?" — the budget, not any branch, is the object.

T2 — Declared Multiplicity versus Silent Multiplicity (measurement). Declared multiple testing has standard corrections; the pathology here is that comparisons are made silently, as analytic choices rather than explicit tests, so the corrections do not apply because no one can see the budget. The failure mode is applying a Bonferroni correction to the reported tests while the unreported tree dwarfs them. Diagnostic: the inferential warrant depends on the size of the tree explored, not the comparisons reported — a counterfactual ("which fraction would have been reported had it come out positive?") that no reader can audit.

T3 — Garden of Forking Paths versus Pre-Planned Single Comparison (temporal). The warrant gap opens when the tree was large and data-dependent — choices made after seeing the data. Even a single declared comparison is contaminated if the path to it forked on the data. The failure mode is treating a data-dependent analysis as though it were pre-planned, so a nominal 5% test runs far hotter. Diagnostic: were the analytic choices fixed before the data were seen, or could the data have steered them? Data-dependence, even with one final test, exercises the hidden budget.

T4 — Collapse-the-Budget versus Reveal-the-Budget (sign/intervention). Every remedy is the same structural move expressed two ways: pre-commitment (registration, holdout) collapses the tree to one path before the data; multiverse/specification-curve reporting reveals the whole distribution after. The failure mode is conflating them or doing neither — reporting one leaf with no pre-commitment and no multiverse. Diagnostic: did the analysis either fix the tree in advance or show the result across all defensible branches? If neither, the silent budget remains uncorrected.

T5 — Framed Warrant Claim versus Bare Structural Skeleton (scopal). The prime imports a reporting-norm context — "researcher," false-positive inflation as inferential failure — that makes the warrant claim meaningful but ties it to scientific/evaluative institutions. The bare structure (silent multiplicity collapsed into a singular declaration) is portable, but stripped of the norm it becomes merely descriptive. The failure mode is applying the warrant critique where no reporting institution sets a standard for what a result must warrant. Diagnostic: is there an institution whose norms the silent multiplicity violates? Without one, the structure travels but the evaluative charge does not.

T6 — Holdout Separation versus Leakage (coupling). Held-out test sets physically separate exploration from evaluation so exploration multiplicity cannot leak in — but the separation fails the moment exploration touches the holdout, even indirectly (tuning on it, peeking, reusing it across projects). The failure mode is believing a holdout protects when it has been silently contaminated. Diagnostic: trace every contact with the evaluation data during exploration — any leakage re-couples the two and reinstates the full silent budget the holdout was meant to exclude.

Structural–Framed Character

Researcher Degrees of Freedom sits at the far framed end of the structural–framed spectrum — framed, aggregate 1.0, with every one of the five diagnostics reading the maximum. It is the most framed entry in its cohort: a prime that imports a scientific-reporting context wholesale, where the inferential-warrant claim that gives it meaning is inseparable from the norms of empirical research practice. There is a portable structural skeleton underneath — silent multiplicity collapsed into a singular declaration — but as the prime is written, every facet carries the inherited frame, and this section defends that reading rather than inflating it toward structural.

vocab_travels is 1.0 because the home lexicon — "researcher," "garden of forking paths," "false-positive inflation," "pre-registration," "comparison budget" — is research-methodology vocabulary that must be translated to reach finance backtests, ML benchmarks, or audits, and even the core term "researcher" presupposes a scientific actor. evaluative_weight is 1.0 because the prime is explicitly normative: false-positive inflation is framed as an inferential failure, a thing gone wrong against a standard of what a reported result ought to warrant, so the construct cannot be stated value-neutrally the way "feedback" or "reaction intermediate" can. institutional_origin is 1.0 because the pattern presupposes a reporting institution — its entire force depends on there being a declared result that public norms expect to be a single pre-planned comparison, against which silent multiplicity is a violation. human_practice_bound is 1.0 because every instance is a human or organizational practice of analysis and reporting; there is no physical or biological substrate in which the pathology runs — it requires an agent who explores a decision tree and selectively declares a leaf. And import_vs_recognize is 1.0 because invoking the prime IMPORTS the whole reporting-norm framework — the notion of a warrant, an auditable counterfactual, an obligation to disclose the budget — rather than merely RECOGNIZING a pattern already wired into an indifferent system. The bare structural core (a flexible decision tree culminating in one declared leaf) is real and portable, and the entry's own text concedes this; but stripped of the reporting norm it becomes merely descriptive, and it is precisely the imported institutional and evaluative context that makes the warrant critique meaningful. On every diagnostic, the prime reads framed, and the aggregate of 1.0 is faithful.

Substrate Independence

Researcher Degrees of Freedom is moderately substrate-independent — composite 3 / 5 on the substrate-independence scale. Its domain breadth is moderate (3): the garden-of-forking-paths pattern — flexible analytic choices culminating in a singular declared result — recurs in statistics and empirical science (where ordinary flexibility can push a nominal 5% false-positive rate above 60%), machine-learning evaluation (test-set tuning, benchmark and prompt selection), financial backtesting, policy evaluation, accounting, and journalism. The structural shape, a branching tree of defensible choices collapsed to one reported path, is genuinely portable, which earns the moderate breadth. What caps the composite is that every instance is an analysis-and-reporting practice presupposing an analyst, a degree of discretion, and an audience that receives a single result — there is no physical or biological substrate where the pattern runs without an interpreting agent. Structural abstraction is therefore mid (3): the skeleton is relational but carries the inherited frame of empirical research conduct. Transfer evidence is the strongest component (4): the forking-paths formalism and its inflation of error rates are concretely documented across statistics, ML evaluation, and backtesting, carrying the same quantitative force. The prime is recognized across these analytic domains but stays anchored to research-and-reporting substrates, which holds it at the moderate band.

  • Composite substrate independence — 3 / 5
  • Domain breadth — 3 / 5
  • Structural abstraction — 3 / 5
  • Transfer evidence — 4 / 5

Relationships to Other Primes

One-hop neighborhood: parents above, mutual partners to the right, children below.ResearcherDegrees of Freedomsubsumption: BiasBias

Parents (1) — more general patterns this builds on

  • Researcher Degrees of Freedom is a kind of, typical Bias

    RDF is a systematic, directional inferential error (false-positive inflation) produced by an un-audited comparison budget — a specialized inferential bias arising at the analysis/reporting stage, distinct from random noise. is-a bias in the inference pipeline.

Path to root: Researcher Degrees of Freedom → Bias

Neighborhood in Abstraction Space

Researcher Degrees of Freedom sits among the more crowded primes in the catalog (36th percentile for distinctiveness): several abstractions describe nearly the same structure, so a description that fits it will tend to fit its neighbors too — transporting it usually means disambiguating within this family rather than landing on it exactly.

Family — Inference & Evidence (26 primes)

Nearest neighbors

Computed from structural-signature embeddings · 2026-06-14

Not to Be Confused With

The most important confusion to clear is with multiple_comparisons_correction, because the two share a statistical root yet differ on the one feature that makes the prime a prime: visibility. Multiple-comparisons handling deals with declared tests — you ran twenty hypotheses, you correct for twenty, with Bonferroni or FDR. Researcher degrees of freedom is precisely the case where the comparisons are undeclared: they are exercised silently as analytic choices (exclusions, transforms, covariates, stopping rules) rather than as explicit tests, so no one — not the reader, often not even the analyst — can count them, and therefore no correction can be applied. The corrections are not wrong; they are inapplicable, because the budget they would correct for is invisible. This is why applying Bonferroni to the reported tests is a characteristic error: it corrects the tip of the iceberg while the submerged tree dwarfs it. The discriminating question is whether the multiplicity is on the page (multiple comparisons) or behind it (researcher degrees of freedom). Conflating them lets an analyst believe a corrected reported set is clean when the silent budget remains entirely uncorrected.

A second genuine confusion is with overfitting, sharpened by the fact that the financial instance of this prime is literally called "backtest overfitting." But the two name failures at different layers. Overfitting is a property of a model: it fits noise in the training data and generalizes poorly, a phenomenon in the model-data relationship that exists even for a single fully-specified analysis. Researcher degrees of freedom is a property of the reporting process: a large analytic tree is explored and one leaf is declared as though pre-planned, inflating the warrant of that singular report. They co-occur — searching thousands of strategy variants (degrees of freedom) produces a model that overfits the price history — but they are separable. A single model can overfit with no analytic multiplicity at all (one specification, too many parameters), and an analytic tree can inflate false positives even when each leaf is a simple, non-overfit test. The tell is whether the failure is "this model learned noise" (overfitting) or "this result was selected from many and reported as one" (researcher degrees of freedom). Treating them as identical leads one to fix the model (regularize) when the real problem is the un-disclosed selection, or vice versa.

A third confusion worth drawing is with the prime's embedding-nearest neighbor, regret, which is almost certainly a spurious similarity but instructive to dispel. Regret is an affective and decision-theoretic concept: the backward-looking valuation of an outcome relative to a forgone alternative, the felt or computed cost of "the road not taken." Researcher degrees of freedom is an inferential concept about silent multiplicity and warrant. The only thing they share is the abstract presence of "roads not taken" (unexplored branches) — but in regret the unchosen branch matters because of the outcome you missed, whereas in researcher degrees of freedom the unreported branches matter because their existence inflates the false-positive rate of the branch you did report. There is no valuation of forgone outcomes in the prime; there is an accounting of an un-auditable comparison budget. A practitioner should not import regret's outcome-comparison machinery here; the relevant counterfactual is "which fraction of the tree would have been reported had it come out positive?", not "how much better could I have done?"

For a practitioner the cuts route to different fixes. If the multiplicity is declared, correct for it (multiple comparisons). If a model learned noise, regularize and validate out-of-sample (overfitting). If a causal common cause lurks, adjust for it (confounding). Researcher degrees of freedom specifically demands making the silent budget visible or pre-committed — registration, holdout, multiverse — because its pathology is exactly the multiplicity that none of the other frames can see.

Solution Archetypes

No catalogued solution archetypes reference this prime yet.