A/B Test Interpretation Protocol¶
Experimentation protocol — instantiates Hypothesis Testing Frame
Reads a live randomized experiment against a pre-declared primary metric and launch criteria, turning the measured difference into a ship, hold, or iterate decision.
An A/B Test Interpretation Protocol is the governance wrapper around an online randomized experiment: it fixes, before traffic is split, which metric decides the launch and what result ships the change — then holds the team to it once the numbers land. Its defining concern is not computing a difference but disciplining what an organization is allowed to conclude from a live experiment where dozens of metrics move at once and everyone has a favored one. What makes this THIS mechanism is the pre-registration of a single primary metric and launch bar plus the explicit accounting for the many other metrics being watched — so that a ship decision reflects the question the team committed to, not whichever chart looked best after the fact.
Example¶
A retail website tests a redesigned checkout button, hypothesizing it will lift completed purchases. Before launch, the growth team writes the decision down: the primary metric is checkout-completion rate; the launch bar is a statistically clear lift with no regression in the guardrail metrics (page-load time, refund rate, revenue per session). Traffic is split 50/50 for two full weeks to cover weekday and weekend behavior.
Results arrive noisy. Completion is up a bit but not past the bar; meanwhile "add-to-cart clicks" — one of fifteen secondary metrics on the dashboard — is up sharply and tempting to headline. The protocol's job is to refuse that substitution: add-to-cart was not the pre-declared decider, and with fifteen secondary metrics in play, one big mover is expected by chance alone. The verdict is iterate, do not ship: the primary metric did not clear the bar, no guardrail broke, and the team ships nothing while it reworks the button. Had it shipped on the add-to-cart spike, it would have launched on a number chosen after seeing the data.
How it works¶
- Pre-register the decider. One primary metric and an explicit launch bar are written down before the experiment starts, alongside the guardrails that can veto a launch.
- Run to plan. Fix the horizon and traffic split up front; do not stop the moment the primary metric looks good, which inflates false positives.
- Account for the metric family. Read secondary metrics as exploratory context, discounted for how many are being scanned, never as the decision.
- Emit a decision, not a chart. Ship, hold, or iterate — the primary result and guardrails together produce one action.
Tuning parameters¶
- Primary vs guardrail set — which single metric decides and which can only veto. A crisp split prevents metric-shopping; too many "primaries" reintroduces it.
- Launch bar — the size of lift required to ship. Higher avoids shipping noise but discards small real wins.
- Experiment horizon — how long to run; longer covers cycles and novelty decay but delays the decision and costs traffic.
- Peeking policy — whether interim looks are allowed and how they are corrected; unmanaged peeking silently multiplies false positives.
When it helps, and when it misleads¶
Its strength is that it converts a dashboard full of movement into a single committed decision, immunizing launches against cherry-picking and post-hoc storytelling. It is the discipline that lets a team trust its own experiment.
Its failure mode is Twyman's law territory: the most eye-catching number on the dashboard is the one most likely to be a fluke or an instrumentation artifact, and a team that lets a surprising secondary metric override its pre-registered primary is usually chasing noise.[n1] The classic misuses are stopping the test the instant the primary metric peeks over the bar, and quietly swapping the primary metric after the read. The guarding discipline is to freeze the primary metric, bar, and horizon before launch, discount the secondary metrics for their number, and let only the committed decision rule fire.
How it implements the components¶
claim_under_test— the specific shippable change and the primary metric it is claimed to move.decision_rule— the ship / hold / iterate rule combining the primary result with guardrail vetoes.multiplicity_context— explicit accounting for the many secondary metrics scanned, so one chance mover is not mistaken for a finding.precommitment_record— the primary metric, launch bar, and horizon recorded before traffic is split.
It does not itself set the numeric evidence_threshold a metric must clear — that operational cut point is Decision Threshold Rule — nor gather and model the sampled test_evidence, which is the work of Null Hypothesis Significance Test; this protocol governs which metric and which claim that machinery is applied to.
Related¶
- Instantiates: Hypothesis Testing Frame — the product-experimentation realization that ties evidence to a launch decision.
- Consumes: Null Hypothesis Significance Test supplies the underlying comparison; Decision Threshold Rule supplies the launch cut.
- Sibling mechanisms: Decision Threshold Rule · Sequential Review Gate · Null Hypothesis Significance Test · Scientific Claim Evaluation Template
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: The mechanism reads a live randomized experiment against a pre-declared primary metric and launch criteria, turning the measured difference into a ship, hold, or iterate decision, so its operative form is a bounded assessment of existing evidence or work.
Independent corroboration: The frozen evidence defines A/B Test Interpretation Protocol as 'Reads a live randomized experiment against a pre-declared primary metric and launch criteria, turning the measured difference into a ship, hold, or iterate decision', so its operative form is Assessment, Review & Assurance.
Nearest alternative: Decision, Gate & Allocation — It evaluates already-produced experiment evidence; ship, hold, or iterate is the resulting disposition.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Pre-registration, randomized comparison, multiplicity control, fixed horizons, and decision thresholds arise directly from experimental design and statistical inference.
Related originating lineages:
- Data Science & Analytics — Large-scale online experimentation supplied the live product metrics, guardrails, and operational interpretation setting.
- Organizational & Management Science — The ship-hold-iterate governance wrapper and commitment against post-hoc metric shopping are organizational decision controls.
Review resolution: The reviewers' classifications are exact; the ambiguity text correctly marks the modern governance wrapper without weakening the statistical primary lineage.
Attribution caveat: The statistical core is clear; the protocol as written is a modern product-experiment governance synthesis rather than a purely statistical test.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Twyman's law holds that any figure that looks interesting or unusual is usually wrong — the more surprising a metric, the more likely it reflects an error, an artifact, or noise rather than a real effect. In experimentation it is the standing warning against headlining the flashiest number on a dashboard. ↩