Small Experiment¶
Evidence probe — instantiates Risk Aversion Calibration
Buys decision-relevant evidence under a strict downside cap, converting a reducible unknown into a signal before any full commitment.
Some caution is not distortion at all — it is honest ignorance, and the cure is not a better argument but a fact. Small Experiment is the move that spends a little exposure to buy that fact: it isolates the one unknown the decision actually turns on, designs the cheapest action that would produce a signal on it, caps the downside of running that action, and reads the result against a pre-committed threshold. Its defining property is that it exists to improve the risk estimate through action — it is an evidence-generation instrument, not a scaled-down version of the real thing. It need not be reversible, and it need not even be real: a throwaway probe, a fake door, a wizard-of-oz mock-up all count, so long as they return a trustworthy signal on the pivotal question under bounded loss. What disqualifies it is running an "experiment" with no learning question, no success criterion, and no rule for what the result changes.
Example¶
A subscription software team is split over whether to raise the entry-tier price by a third. One camp fears mass churn; the other fears leaving money on the table. Nobody can settle it by arguing, because the disagreement is about a number — how price-sensitive new signups actually are — that no one has measured. Small Experiment names that number as the pivotal unknown and designs the cheapest read on it: route two percent of new-signup traffic to the higher price for two weeks, hold everything else constant, and pre-commit to the decision rule before seeing data — say, "if conversion on the test arm drops by more than a fifth, the increase is off the table."
The downside is capped by construction: only new visitors in a thin traffic slice see the new price, existing customers are untouched, and the test ends on a fixed date. Two weeks later the team has what no meeting could give them — an actual conversion signal on the exact question in dispute — and the argument collapses into a reading of the result rather than a clash of intuitions.
How it works¶
- Isolate the pivotal unknown. Find the single question whose answer would move the decision, and ignore the unknowns that would not. An experiment that tests everything tests nothing.
- Design the cheapest sufficient probe. Pick the smallest, fastest action that returns a trustworthy signal on that one question — a traffic slice, a mock, a single-market trial — not a rehearsal of the full launch.
- Cap the exposure. Bound the probe by scope, audience, budget, and time so that the worst case is a known, tolerable loss rather than the full downside.
- Pre-commit the read. Fix the success criterion and the next-decision rule before the data lands, so the result updates the estimate instead of being reinterpreted to fit the answer people wanted.
Tuning parameters¶
- Exposure budget — how much you are willing to lose to learn. A bigger probe buys a cleaner signal but spends more and risks becoming the commitment it was meant to precede.
- Signal fidelity vs. speed — a fake-door mock returns a fast, noisy read; a real limited launch returns a slower, truer one. Match fidelity to how expensive being wrong would be.
- Pre-registration strictness — how firmly the decision rule is locked before data arrives. Firmer rules blunt hindsight rationalization but can lock you into a threshold that later looks crude.
- Stopping rule — a fixed end date versus a run-until-significant design. Fixed dates prevent fishing; open-ended runs risk peeking and false positives.
When it helps, and when it misleads¶
Its strength is that it turns reducible uncertainty into evidence at a price you choose, which is the honest answer to fear that is really just missing information. It is a safe-to-fail probe: designed so that a bad result costs little and teaches much.[n1] Where framing only re-describes a risk, a Small Experiment actually shrinks it.
It misleads when the uncertainty is not reducible by a small probe — when the thing you fear only appears at scale, over long horizons, or in the tail — because a clean small-sample signal can then flatter a decision the full rollout will punish. The classic misuse is treating a tiny, unrepresentative test as a verdict: a two-week, two-percent read says little about a year-long, whole-market outcome. The guarding discipline is to state, before running, exactly what the probe cannot tell you, and to size the exposure to match how far the signal will really generalize.
How it implements the components¶
Small Experiment fills the evidence-and-exposure slots, and only those:
objective_risk_estimate— its whole purpose: it produces a measured signal on the pivotal unknown, replacing an imagined likelihood with an observed one.downside_protection— the probe is exposure-capped by scope, audience, budget, and time, so learning happens under a bounded, known worst case.
It does not itself select the go/hold/decline posture (hedge_or_commitment_choice belongs to the deciding mechanisms). Its nearest twin, Reversible Pilot, is defined by its reversibility_assessment — a pilot must be a real deployment that can be rolled back; a Small Experiment need be neither reversible nor real, because what defines it is the evidence signal, not the retreat path.
Related¶
- Instantiates: Risk Aversion Calibration — a Small Experiment supplies the fresh evidence that lets calibration proceed on facts rather than fear.
- Consumes: Downside Cap — the probe's exposure limit is a cap applied to the experiment itself.
- Sibling mechanisms: Risk Framing · Downside Cap · Hedging or Insurance · Reversible Pilot · Expected-Value Review · Opportunity Cost Reflection · Risk Matrix
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Small Experiment operates as an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation because it buys decision-relevant evidence under a strict downside cap, converting a reducible unknown into a signal before any full commitment.
Independent corroboration: The frozen evidence defines Small Experiment as 'Buys decision-relevant evidence under a strict downside cap, converting a reducible unknown into a signal before any full commitment', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Convergent development
Present-day reach: Universal
Rationale: A bounded intervention designed to buy decision-relevant evidence is an experiment, with scale limiting downside rather than changing its inferential lineage. GOV.UK pilot guidance requires testable learning and evidence before broader commitment.
Related originating lineages:
- Behavioral Economics — behavioral_economics contributes a distinct disciplinary practice to this mechanism's defining operation—Buys decision-relevant evidence under a strict downside cap, converting a reducible unknown into a signal before any full commitment—without displacing the selected primary historical lineage.
- Engineering & Design — Small prototypes validate technical assumptions under a damage cap.
- Innovation & Entrepreneurship — Lean experiments test uncertain value before major commitment.
- Operations Research — operations_research contributes operations research, optimization, and queueing analysis to this mechanism's defining operation—Buys decision-relevant evidence under a strict downside cap, converting a reducible unknown into a signal before any full commitment—without displacing the selected primary historical lineage.
- Organizational & Management Science — Governed pilots convert findings into staged decisions.
Review resolution: The blind reviewers disagree on primary lineage (statistics_experimental_design versus innovation_entrepreneurship). Authoritative or primary research supports statistics_experimental_design as the best historical origin: A bounded intervention designed to buy decision-relevant evidence is an experiment, with scale limiting downside rather than changing its inferential lineage. GOV.UK pilot guidance requires testable learning and evidence before broader commitment. The cited GOV.UK Service Manual, How the Beta Phase Works; GOV.UK, Testing and Piloting Services Guidance directly supports the mechanism's defining operation. All independently supported contributing domains are retained without an arbitrary cap. origin_mode=convergent records lineage, while domain_reach=universal records later applicability separately from provenance.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
Notes¶
[n1] A safe-to-fail probe — a term from the Cynefin framework for complex problems — is a small action designed so that failure is survivable and informative, run to reveal how a system responds rather than to implement a predetermined solution. It is the operating logic behind a well-formed Small Experiment. ↩