Repeated Significance Tests on Accumulating Data.¶
Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated Significance Tests on Accumulating Data. Journal of the Royal Statistical Society, Series A, 132(2), 235-244.
Cited by¶
4 citations across 4 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Termination Condition
- Termination condition supports several portable inferences. Halting requires a separate argument: proving that an iterative process terminates is structurally distinct from proving that it does the right thing while iterating, and conflating the two is a recurring engineering bug. Monotone-progress plus bounded-state implies termination: if each step monotonically changes a state variable that lives in a bounded set, termination is forced, which is the underlying form of nearly every classical termination proof — loop variants, well-founded recursion, decreasing measure functions. No-progress conditions are dangerous near optima: a stopping rule based on "the objective stopped improving" can halt arbitrarily close to or far from the optimum depending on the landscape, with the cross-domain symptom of solvers converging to wrong answers and early-stopped models acquiring avoidable bias. Optional stopping inflates false positives: a rule allowed to look at the data and choose when to stop violates fixed-design statistical guarantees, the same structural fact appearing in p-hacking, mid-game concession in negotiation, and early stopping for non-inferiority.
This sourceFoundational demonstration that repeated significance testing on accumulating data inflates the Type I error above the nominal level — the basis for stopping rules in sequential testing.
Borderline supportVerified against the publisher's abstract
The work's own abstract states that repeated significance testing on accumulating data raises the probability of a significant result above the nominal level.
“If the null hypothesis is true and if each significance test is performed at the same nominal level, the probability that at some stage or another the test criterion is may be substantially greater than the nominal value.”
Two runs of the same check disagreed on this citation; the later verdict is shown and the citation is queued for review.
- Termination condition supports several portable inferences. Halting requires a separate argument: proving that an iterative process terminates is structurally distinct from proving that it does the right thing while iterating, and conflating the two is a recurring engineering bug. Monotone-progress plus bounded-state implies termination: if each step monotonically changes a state variable that lives in a bounded set, termination is forced, which is the underlying form of nearly every classical termination proof — loop variants, well-founded recursion, decreasing measure functions. No-progress conditions are dangerous near optima: a stopping rule based on "the objective stopped improving" can halt arbitrarily close to or far from the optimum depending on the landscape, with the cross-domain symptom of solvers converging to wrong answers and early-stopped models acquiring avoidable bias. Optional stopping inflates false positives: a rule allowed to look at the data and choose when to stop violates fixed-design statistical guarantees, the same structural fact appearing in p-hacking, mid-game concession in negotiation, and early stopping for non-inferiority.
Mechanisms¶
- Alpha-Spending Plan
- It is the disciplined answer to the temptation that quietly ruins repeated testing — optional stopping, the practice of continuing to collect and re-test data until the result crosses significance, which can drive the true error rate arbitrarily high.
This sourceRepeatedly retesting accumulating data without adjustment inflates Type I error and can eventually produce an arbitrarily significant result under the null.
- It is the disciplined answer to the temptation that quietly ruins repeated testing — optional stopping, the practice of continuing to collect and re-test data until the result crosses significance, which can drive the true error rate arbitrarily high.
- Champion / Challenger Threshold Test
- Its classic misuse is peeking — watching the running comparison
This sourceShows that repeatedly testing accumulating data raises the probability of a false positive under the null.
- Its classic misuse is peeking — watching the running comparison
- Rolling Batch Size A/B Test
- Calling a winner too early — glancing at the dashboard and stopping the moment it looks good — inflates false positives badly; repeatedly peeking at an accumulating result without correcting for it is a well-known way to fool yourself.
This sourceShows that repeatedly testing accumulating data at a fixed significance level raises false-positive risk above the nominal level.
- Calling a winner too early — glancing at the dashboard and stopping the moment it looks good — inflates false positives badly; repeatedly peeking at an accumulating result without correcting for it is a well-known way to fool yourself.
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 1 of 4 citations: 1 borderline. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:289aa3aba27f · see in the full table