A/B Test Readout¶
Experiment readout — instantiates Variation Consolidation and Feature Selection
Reads out a controlled A/B experiment — the measured lift, its confidence, and the pre-registered metric — to declare which variant actually won.
A/B Test Readout is the step that turns a finished controlled experiment into a defensible statement about which variant performed better. It does not run the rollout and it does not decide the variant's organizational fate; its single job is to read the experiment honestly — measure the outcome each arm produced, express the difference with its uncertainty, and check it against the metric that was declared before the test began. Its defining move is that a pre-registered primary metric and an explicit confidence statement are the arbiters, so "B looks better" becomes "B lifted the primary metric by X ± Y, clearing (or failing) the decision threshold." It reads out the evidence; it stops before acting on it.
Example¶
A food-delivery app is testing two checkout layouts: A, the current multi-step flow, and B, a one-tap reorder. Users are randomized 50/50 for three weeks. The primary metric is fixed before launch — completed-order rate — along with two guardrail metrics, refund rate and support contacts, that must not degrade. When the window closes, the readout reports that B lifted completed-order rate by 1.8% (95% confidence interval 0.9–2.7%) while both guardrails stayed flat. Because the metric and threshold were set in advance and the interval clears zero, the readout declares B the winner — and it flags that the effect is small enough that the team should watch for a novelty bump that fades. It does exactly that much. Whether to ship B, and how, is a separate decision the readout hands off rather than makes.
How it works¶
What distinguishes an honest readout from "eyeballing the dashboard":
- Fix the metric and threshold first. The primary metric, minimum detectable effect, and decision rule are pre-registered, so the readout cannot shop the data for a flattering slice after the fact.
- Measure both arms on the same population and window. Each variant's outcome and guardrails are captured under identical conditions, which is what makes the difference causal rather than correlational.
- Express the difference with uncertainty. The output is an effect size with a confidence interval, never a bare point estimate — the width of the interval is part of the verdict.
- Check against the decision threshold. Statistical significance and practical magnitude are tested, plus a sanity check for sample-ratio mismatch that would signal a broken experiment.
Tuning parameters¶
- Confidence level / MDE — how strict the bar is. Tighter thresholds cut false wins but demand more traffic and longer runs.
- Guardrail set — how many protective metrics can veto a headline win. More guardrails catch collateral damage but make a clean "win" rarer.
- Test horizon — how long the experiment runs. Longer filters novelty effects and weekly cycles but delays the call.
- Segmentation depth — whether the readout reports one overall effect or many per-segment effects. Segments reveal heterogeneity but multiply the chance of a spurious "significant" cell.
- Stopping rule — fixed-horizon versus sequential testing. Peeking at results and stopping early without a sequential correction silently inflates the error rate.
When it helps, and when it misleads¶
Its strength is that it replaces opinion and seniority with a like-for-like causal estimate carrying an honest uncertainty band — the difference between "we think B is better" and "B beat A by a measured, bounded amount on the metric we agreed mattered."
Its signature failure is the peeking problem: checking an experiment repeatedly and stopping the moment it crosses significance turns a 5% false-positive rate into something far higher, so a random blip gets crowned as a real effect.[n1] Two more shadow it. A "too good to be true" result usually is — an instrumentation or randomization bug (Twyman's-law territory) inflating the lift. And the measured metric can be gamed: a variant that boosts the tracked number while quietly degrading something unmeasured reads as a winner. The discipline is to fix metric, threshold, and horizon before launch, keep guardrails watching the unmeasured harms, and never stop early without a sequential correction.
How it implements the components¶
A/B Test Readout fills the evidence-and-comparison slice of the archetype — the parts that establish which variant performed better, not what to do about it:
performance_evidence_capture— it measures each arm's outcome and guardrail metrics under identical conditions; that measured behavior is the experiment's evidence.variant_comparison_frame— it renders treatment against control side by side with effect size and confidence interval, making both the difference and its uncertainty legible.selection_criterion_set— the pre-registered primary metric and decision threshold are the explicit standard the verdict is judged against, set before the contenders' numbers are known.
It reads the experiment but does not act on it: it does not make the retain / adapt / retire call or record a retirement — selective_retention_rule, retirement_or_hold_record — that's Post-Pilot After-Action Review; nor route a winner into standard practice — consolidation_pathway — that's Best-Practice Harvesting Workshop.
Related¶
- Instantiates: Variation Consolidation and Feature Selection — supplies the comparative evidence the rest of the consolidation depends on.
- Sibling mechanisms: Feature-Flag Graduation Review · Post-Pilot After-Action Review · Best-Practice Harvesting Workshop · Model Feature Selection Protocol · Champion–Challenger Evaluation · Multi-Criteria Selection Rubric · Merge and Deprecation Plan
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: The mechanism reads out a controlled A/B experiment — the measured lift, its confidence, and the pre-registered metric — to declare which variant actually won, so its operative form is a bounded assessment of existing evidence or work.
Independent corroboration: The frozen evidence defines A/B Test Readout as 'Reads out a controlled A/B experiment — the measured lift, its confidence, and the pre-registered metric — to declare which variant actually won', so its operative form is Assessment, Review & Assurance.
Nearest alternative: Analysis, Modeling & Optimization — Its primary act is judging experiment evidence against preregistered criteria, with effect estimation supporting that judgment.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: A randomized treatment-control readout using a pre-specified estimand, effect size, confidence interval, and stopping rule is a canonical product of experimental design and statistics.
Related originating lineages:
- Data Science & Analytics — Production analytics and online experimentation supplied the modern instrumented A/B setting and segment and guardrail reporting practice.
Review outcome: Independent reviewer agreement; high confidence.
Notes¶
The readout declares a winner, not a keeper. A variant can beat the control on the chosen metric and still be the wrong thing to standardize — if the metric is a weak proxy, or the win is too small to justify the maintenance cost. Keeping the readout an input rather than a verdict is what lets a downstream review (Post-Pilot After-Action Review, Feature-Flag Graduation Review) weigh the win against everything the experiment did not measure.
[n1] The peeking problem (optional stopping) — inspecting an ongoing experiment and stopping as soon as it reaches significance inflates the false-positive rate well above its nominal level, because each look is another chance for noise to cross the line. The standard corrective is a fixed horizon or a sequential test designed to remain valid under continuous monitoring. ↩