Trustworthy Online Controlled Experiments¶
Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.
Cited by¶
13 citations across 13 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Comparative Method
- Mapping Comparative Method into engineering A/B testing and benchmarking:
This sourceOperational treatment of A/B testing as the engineering realization of comparative-method logic; covers selection bias, statistical power, and unit-of-analysis problems — supports both the Knowledge-Transfer mapping and the Applied/Industry example.
Supported in partVerified against the publisher's abstract
Supplies the A/B-testing half of the mapping: its abstract presents online controlled experiments as the scientific-method way to test hypotheses, but says nothing of comparative method or benchmarking.
“This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests.”
- Mapping Comparative Method into engineering A/B testing and benchmarking:
- Control Sample
- And in software it is shadow deployments and canary releases holding most traffic on the baseline.
This sourceHoldout/control arms, randomized assignment, and concurrent measurement in online experimentation; canary and shadow deployments.
Supported in partVerified against the publisher's abstract
The book's abstract backs controlled online A/B experiments but says nothing about manufacturing control charts or fMRI baseline subtraction, the claim's other two arms.
“This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests.”
- And in software it is shadow deployments and canary releases holding most traffic on the baseline.
- Factorial Design
This sourceThe modern reference on online controlled experiments / A/B testing, including multivariate (factorial) testing on digital platforms.
- Intervention
- The recognition that randomized assignment severs all incoming dependencies transfers verbatim from clinical RCTs to software A/B testing to chaos engineering: the same prime explains why A/B-test design requires independence of assignment from pre-treatment characteristics, why fault injection rather than passive monitoring is required for reliability claims, and why these techniques identify causal mechanisms that no correlational analysis of production logs can.
This sourceEstablishes that randomized assignment in A/B tests and canary deployments severs treatment from pre-treatment characteristics, identifying causal effects production logs cannot.
Supported in partVerified against the publisher's abstract
The abstract frames A/B tests as controlled experiments for evaluating hypotheses; it says nothing of chaos engineering, fault injection or clinical RCTs.
“Use the scientific method to evaluate hypotheses using controlled experiments”
- The recognition that randomized assignment severs all incoming dependencies transfers verbatim from clinical RCTs to software A/B testing to chaos engineering: the same prime explains why A/B-test design requires independence of assignment from pre-treatment characteristics, why fault injection rather than passive monitoring is required for reliability claims, and why these techniques identify causal mechanisms that no correlational analysis of production logs can.
- Proxy–Target Fidelity
- The prime's management response is the industry's: pair the optimized proxy with guardrail metrics the optimizer is not allowed to game (long-horizon retention, survey-based satisfaction, complaint rates), hold out distributions to detect collapse, and treat a rising proxy with a flat or falling guardrail as evidence of fidelity erosion rather than success.
This sourceDefines guardrail (counter-) metrics paired with the optimized OEC to detect harm the primary proxy cannot see, e.g., protecting retention/quality while optimizing an engagement metric.
Supported in partVerified against the publisher's abstract
The publisher abstract establishes only this book's scope — defining key metrics and an OEC and testing result trustworthiness — and never names guardrail metrics, hold-out distributions, or fidelity-erosion diagnosis.
“Learn how to • Use the scientific method to evaluate hypotheses using controlled experiments • Define key metrics and ideally an Overall Evaluation Criterion • Test for trustworthiness of the results and alert experimenters to violated assumptions”
- The prime's management response is the industry's: pair the optimized proxy with guardrail metrics the optimizer is not allowed to game (long-horizon retention, survey-based satisfaction, complaint rates), hold out distributions to detect collapse, and treat a rising proxy with a flat or falling guardrail as evidence of fidelity erosion rather than success.
- Randomization
- However, in pragmatic real-world implementations (open-label trials, mobile-app experiments with visible assignment), maintaining full concealment is impossible or counterproductive
This sourceKohavi A/B testing randomized controlled experiments online platforms.
- However, in pragmatic real-world implementations (open-label trials, mobile-app experiments with visible assignment), maintaining full concealment is impossible or counterproductive
- Variance Bounds Selection Response
- A consumer-products company runs a mature A/B-testing programme and watches its headline conversion metric improve at three percent per quarter for eight quarters, then decay to half a percent at unchanged variant volume and unchanged statistical rigour.
This sourceA/B-testing improvement rate set by the variance of the variant population; low-variance programmes plateau.
- A consumer-products company runs a mature A/B-testing programme and watches its headline conversion metric improve at three percent per quarter for eight quarters, then decay to half a percent at unchanged variant volume and unchanged statistical rigour.
- Washout Failure
- The identical structure governs online A/B testing: re-exposing the same user pool to a new variant before behavioral adaptation to the old variant has decayed contaminates the new variant's lift estimate, and the fix is a washout gap matched to the adaptation-decay timescale or a between-users split so no user sees both.
This sourceCovers carryover/residual and novelty/primacy effects in online experiments and remedies such as washout periods and between-users assignment so no user sees both variants.
Supported in partVerified against the publisher's abstract
The book's own description confirms it treats carryover effects as a pitfall of online controlled experiments, but says nothing about washout gaps, adaptation-decay timescales, or between-users splits.
“Avoid pitfalls like carryover effects and Twyman's law”
- The identical structure governs online A/B testing: re-exposing the same user pool to a new variant before behavioral adaptation to the old variant has decayed contaminates the new variant's lift estimate, and the fix is a washout gap matched to the adaptation-decay timescale or a between-users split so no user sees both.
Domain-specific¶
- Pivot Thrashing
- A strategic direction generates evidence — user behaviour, retention data, willingness-to-pay signals, technical feasibility results — over a characteristic time horizon that depends on sales-cycle length, product-iteration cycle, and the statistical power requirements
This sourceEstablishes that an experiment's required duration is set by the statistical power needed to detect the effect on the chosen metric.
- A strategic direction generates evidence — user behaviour, retention data, willingness-to-pay signals, technical feasibility results — over a characteristic time horizon that depends on sales-cycle length, product-iteration cycle, and the statistical power requirements
Mechanisms¶
- A/B Test Interpretation Protocol
- Its failure mode is Twyman's law territory: the most eye-catching number on the dashboard is the one most likely to be a fluke or an instrumentation artifact, and a team that lets a surprising secondary metric override its pre-registered primary is usually chasing noise.
This sourceExplains Twyman’s law as a warning that unusually striking experimental results are especially likely to reflect data-quality, instrumentation, or analysis problems.
- Its failure mode is Twyman's law territory: the most eye-catching number on the dashboard is the one most likely to be a fluke or an instrumentation artifact, and a team that lets a surprising secondary metric override its pre-registered primary is usually chasing noise.
- Automated A/B Balance Dashboard
- Sample ratio mismatch tests are extraordinarily sensitive at large n, so trivial and harmless logging asymmetries can trip nuisance alarms, and — worse — a green dashboard invites complacency even though a perfect 50/50 split says nothing about whether the right covariates are balanced.
This sourceShows that sample-ratio mismatch tests become highly sensitive at large sample sizes and that matching the planned treatment allocation does not by itself establish balance on relevant covariates.
- Sample ratio mismatch tests are extraordinarily sensitive at large n, so trivial and harmless logging asymmetries can trip nuisance alarms, and — worse — a green dashboard invites complacency even though a perfect 50/50 split says nothing about whether the right covariates are balanced.
- Guardrail Dashboard
- The guarding discipline, borrowed from online experimentation, is to keep guardrail metrics read-only and separate from the optimized objective — genuine "do no harm" constraints rather than second objectives — and to hang a real decision or review on the board so the red actually stops something.
This sourceTreats guardrail metrics as do-no-harm checks that constrain an experiment decision and can block launch when an important non-objective metric degrades.
- The guarding discipline, borrowed from online experimentation, is to keep guardrail metrics read-only and separate from the optimized objective — genuine "do no harm" constraints rather than second objectives — and to hang a real decision or review on the board so the red actually stops something.
- Quality Guardrail Gate
- It is the operational form of what experimentation teams call guardrail metrics — measures a change must not harm even while it improves the primary one.
This sourceDefines guardrail metrics as measures that must not deteriorate while an experiment improves its primary evaluation criterion.
- It is the operational form of what experimentation teams call guardrail metrics — measures a change must not harm even while it improves the primary one.
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 5 of 13 citations: 5 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Links previously used in the corpus¶
Before the registry existed this work was also linked 2 other ways.
- https://doi.org/10.1017/9781108653985 ×9
- https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26382EB0CB2DC2DB4ED3141769F1 ×1
Registry ID ref:ad40f7c11e4f · see in the full table