Peeking at A/B Tests¶
Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2017). Peeking at A/B Tests: Why It Matters, and What to Do About It. ACM, 1517-1525.
Cited by¶
1 citation across 1 artifact.
Each citation links to the sentence it supports in the citing article.
Mechanisms¶
- A/B Test Readout
- Its signature failure is the peeking problem: checking an experiment repeatedly and stopping the moment it crosses significance turns a 5% false-positive rate into something far higher, so a random blip gets crowned as a real effect.
This sourceShows that repeatedly peeking at an A/B test and stopping at significance can sharply inflate false positives, turning null fluctuations into apparent effects.
- Its signature failure is the peeking problem: checking an experiment repeatedly and stopping the moment it crosses significance turns a 5% false-positive rate into something far higher, so a random blip gets crowned as a real effect.
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Not recorded. The single citation of this work carries no recorded support check.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:24970ae13e18 · see in the full table