Tensions in Practice: Pre-specified tests in tension with discovery¶
A study with an unexpected pattern
A planned statistical test declares its question and decision rule before seeing the results. Exploration instead follows patterns found in the data; that can reveal worthwhile new questions, but a question chosen because it fits these data cannot simply borrow the planned test’s evidential standing. Compare keeping only the planned path with adding a separate discovery path that leads to a new, pre-specified test on fresh data.
Keep the test interpretable
Preserve the declared analysis and its conditional error-control claim.
Follow unexpected patterns
Let surprising observations generate questions worth investigating.
Why these aims pull against each other
Following a surprise uses the data to choose the question. Treating that selected question as if it had been fixed in advance hides the selection step; refusing to explore loses a source of discovery.
Choose an arrangement to see what changes and what remains difficult.
Arrows show which data and decisions support which analysis. A fresh test may or may not support the new claim.
What this choice protects
What it costs
When it fits
Compare the arrangements
Planned test only
Apply the declared test and record unexpected patterns without pursuing a new analysis here.
- What it protects
- The planned analysis remains distinguishable from choices made after seeing the data.
- What it costs
- Potentially useful leads remain unexplored within this workflow.
- When it fits
- Fits a narrowly confirmatory task; it need not forbid later work by someone else.
Illustration note: This deliberately restricted path is an editorial contrast, not a claim that pre-specification requires ignoring surprises.
Separate discovery path
Run the original planned test. Label a second analysis exploratory, use it to choose a new question, then fix that question and its analysis before collecting fresh confirmation data.
- What it protects
- Discovery can proceed without presenting a data-chosen question as pre-specified for the same data.
- What it costs
- The second study costs time and data; a promising lead may fail or remain untested.
- When it fits
- The fresh data must be suitable for the new question, and the new test still needs its assumptions and analysis discipline.
Illustration note: The two-stage path illustrates the source’s subsequent confirmatory study; it does not make exploration erroneous or guarantee confirmation.
What this illustration does—and does not—establish
Hypothesis Testing (Null vs. Alternative): Pre-specification discipline versus flexibility for real-world data patterns explicitly distinguishes exploratory hypothesis generation from subsequent confirmation. The restricted first arrangement and the two-lane study workflow are editorial choices; no success rate or mandatory study design is inferred.
- Fresh data do not cure a bad question, unsuitable measurement, violated assumptions, or selective reporting.
- A test result is not the probability that a hypothesis is true, a measure of practical importance, or automatic evidence of causation.
- This depicts conventional pre-specified confirmation. It is not a taxonomy of every valid adaptive or selective-inference method.
Source entries
Hypothesis Testing (Null vs. Alternative)
The source’s Pre-specification discipline versus flexibility for real-world data patterns passage supplies this contextual tension. The arrangements below are bounded editorial illustrations, not an additional empirical finding.
T1 — Pre-specification discipline versus flexibility for real-world data patterns.
T1 — Pre-specification discipline versus flexibility for real-world data patterns. The inferential warrant of hypothesis testing depends on pre-specification. Real-world research often discovers unexpected patterns that motivate post-hoc analyses; these analyses can be scientifically valuable but cannot maintain the pre-specified inferential warrant. Pre-registration, registered reports, and hold-out data for confirmatory analysis address this tension by distinguishing confirmatory (pre-specified) from exploratory (post-hoc) analyses. Mature practice pre-specifies confirmatory tests, reports exploratory analyses transparently as exploratory, and treats exploration as hypothesis generation for subsequent confirmatory studies.
What the operation commits to
The test institutes a particular epistemic discipline: pre-specify a falsifiable claim, pre-specify the evidential threshold for revision, gather data, and apply the rule without post-hoc adjustment of hypotheses or thresholds.
Testing does not measure practical importance
- Not a measure of effect size or practical importance. A statistically significant result can be trivially small in magnitude; a non-significant result can reflect either small effect or insufficient power.