Skip to content

Controlled Pilot

Bounded trial method — instantiates Response Repertoire Expansion

Exposes a newly-added response to a bounded slice of real conditions before wide reliance, so its readiness, risks, and actual effectiveness are proven on small stakes.

A Controlled Pilot takes a candidate new response live under real but deliberately bounded conditions — one site, one segment, one shift — to learn whether it actually works before the whole system commits to it. Its distinctive move is testing in reality with a contained blast radius: it confirms the enablement is genuinely in place, reviews what could go wrong at small scale, and gathers hard effectiveness evidence. The bound is the whole point — the exposure is real enough to surprise you and small enough that the surprise is survivable.

Example

A retail chain wants to add a response to a fraud pattern — serial "wardrobing" returns — giving store managers authority to flag and decline suspected cases. Rolling that to 1,200 stores untested risks alienating honest customers at scale. Instead they pilot it in three stores for six weeks. First a readiness check: do those managers have the flagging tool, the training, and the written authority? Then a risk review: what is the false-positive harm, and what is the customer-recovery path when an honest return is wrongly declined? Then they measure — fraud caught, honest returns declined in error, manager time spent. The pilot shows the response cuts the target fraud materially but needs a softer decline script to curb false positives — a fix made cheaply across three stores instead of expensively across 1,200.

How it works

The pilot is bounded real-world exposure used to buy evidence. It confirms the option is genuinely operational rather than merely designed, contains the downside through limited scope and a rollback path, and produces the effectiveness data the full-rollout decision hangs on. What distinguishes it from a rehearsal is that the cases are real and consequential; what distinguishes it from a full launch is that the consequences are capped.

Tuning parameters

  • Scope of exposure — how small the live slice is. Smaller is safer and cheaper but gives a weaker signal and takes longer to accumulate cases.
  • Duration — long enough to see real and rare cases, short enough to decide. Too short misses the tail; too long delays the rollout it should inform.
  • Success criteria — the thresholds, set in advance, that mean roll out, revise, or kill. Pre-committing them blocks a motivated reading of the results.
  • Reversibility — how cleanly the pilot can be rolled back; more containment licenses a bolder test.

When it helps, and when it misleads

Its strength is converting an argument about whether a new response will work into evidence, cheaply and reversibly — a bounded pilot's real worth is the value of the information it buys.[1] Its failure modes: pilot conditions may not generalize, because the keen stores chosen to pilot are not the average store (a selection effect); short pilots miss rare cases; and the extra attention on pilot staff can flatter results. The classic misuse is running the pilot to justify a rollout already decided — declaring success regardless of what the data shows. The discipline is to pre-register the success criteria and to pick representative, not flattering, pilot conditions.

How it implements the components

  • resource_readiness_check — confirms the response is genuinely operational (tools, training, authority, access) before it is exposed to real cases.
  • risk_review — surfaces the downside of the live test and the containment and rollback path that keep it bounded.
  • effectiveness_feedback_signal — the hard outcome evidence from real cases that the rollout decision depends on.

It does not author the response it exposes — specification belongs to Exception Handling Playbook and the authoring mechanisms — and it does not decide, from many events over time, which gaps to open; that is After-Action Repertoire Review.

Notes

A pilot proves a response works; it does not, by itself, make it a maintained part of the repertoire. Whether the option graduates — gaining an owner, monitoring, and a retirement rule — is a separate decision. A successful pilot that is never formalized quietly evaporates once the pilot team's attention moves on.

References

[1] In decision analysis, the worth of a test is the expected value of information — how much better the eventual decision becomes for knowing the result. A pilot is justified when the decision it sharpens is valuable and uncertain enough to repay the bounded exposure. ↩