Post-Pilot After-Action Review¶
Review process — instantiates Variation Consolidation and Feature Selection
A structured review of pilot results used to decide retention, adaptation, or retirement.
Post-Pilot After-Action Review is a structured retrospective, held after a pilot has ended, that converts what the pilot did and taught into an explicit fate — retain and scale, adapt and re-run, or retire — and records that decision together with the reasoning and the conditions under which it could be revisited. Its defining move is that it is a learning-and-decision review of a completed, real-world pilot: its output is not a statistical winner but a governed disposition plus a durable record of why, so a retired pilot leaves a reactivation trail and a scaled one leaves an auditable rationale. It decides and records; it does not run the controlled measurement and it does not operate a live release.
Example¶
A school district piloted a new math curriculum in four of its thirty schools for a year. When the year ends, the after-action review convenes the pilot teachers, their principals, and the curriculum office. Working from the pilot's test-score data and the teachers' lived experience, the review reaches a component-level decision: retain-and-adapt — scale the curriculum district-wide, but drop the expensive adaptive-software component that teachers found added little over paper practice. It records that retirement as a hold rather than a hard kill, noting the condition under which the software could be reconsidered (if a substantially cheaper license appears). And it logs the full rationale linking each call back to what the pilot actually showed, so next year's rollout team can see why the curriculum was kept and the software shelved. The deliverable is a disposition plus a reactivation-ready record — not a p-value.
How it works¶
- Convene after the pilot ends, with the people who ran it. The review works from completed results and firsthand experience, not from a live experiment still in flight.
- Work from evidence already captured. It synthesizes the pilot's outcomes and lessons; it consumes measurement rather than generating it.
- Decide the fate explicitly. Retain, adapt-and-re-run, or retire — a stated selective-retention call, made component by component where the pilot was mixed.
- Record holds and log the rationale. Retired or parked elements are archived with reactivation conditions, and the reasoning is written down so a future team can audit and revisit it.
Tuning parameters¶
- Evidence bar — how much proof is required before "retain." A higher bar suits high-stakes or irreversible scaling; a lower one keeps promising-but-unproven work alive.
- Decision granularity — a single whole-pilot verdict versus a component-by-component disposition. Finer granularity salvages the good parts of a mixed pilot but takes longer.
- Hold versus hard-retire — whether rejected elements are archived reactivatable or killed outright. Holds preserve optionality; hard retirement reduces clutter.
- Rationale depth — a one-line reason versus a full lineage back to the evidence. Depth makes later revision possible but costs writing effort.
- Participation breadth — the core pilot team versus a wide stakeholder circle. Breadth builds buy-in and surfaces blind spots but dilutes focus.
When it helps, and when it misleads¶
Its strength is that it closes the pilot loop: it forces a positive anecdote to become a decision, so learning turns into scaled practice or an honest retirement instead of a pilot that quietly persists or fades. It is a direct descendant of the U.S. Army's after-action review, whose whole purpose was to convert an event into transferable lessons.[n1]
Its failure modes are cognitive and political. Hindsight bias lets the review rationalize whatever the sponsor already wanted, reading the evidence to confirm the preferred outcome and calling it learning.[n2] And a review run as a blame exercise suppresses the very failures it most needs to surface, so the lessons that matter never make it into the record. The guarding discipline is to fix the decision criteria before looking at results, keep the review blameless so honest failures come out, and always record a rationale a skeptic could check.
How it implements the components¶
Post-Pilot After-Action Review fills the decision-and-record slice of the archetype — the machinery that turns a finished pilot's learning into a governed, revisable disposition:
selective_retention_rule— it makes the explicit retain / adapt / retire call that converts the pilot's evidence and experience into action.retirement_or_hold_record— it records retired or parked elements together with the conditions under which they could be reactivated, so nothing is killed silently.lineage_and_rationale_log— it logs the reasoning that links the disposition back to the pilot, keeping the decision auditable and open to future revision.
It decides and records but does not measure or operate: it runs no controlled statistical comparison — variant_comparison_frame — that's A/B Test Readout; and it provides no reversible switch or compatibility check for a live release — rollback_or_reactivation_option, compatibility_constraint — that's Feature-Flag Graduation Review.
Related¶
- Instantiates: Variation Consolidation and Feature Selection — this review is the archetype's retain / adapt / retire decision for a completed pilot, with its lineage.
- Consumes: A/B Test Readout or field instrumentation supplies the measured results the review deliberates on.
- Sibling mechanisms: A/B Test Readout · Feature-Flag Graduation Review · Best-Practice Harvesting Workshop · Model Feature Selection Protocol · Multi-Criteria Selection Rubric · Merge and Deprecation Plan · Champion–Challenger Evaluation
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: The review synthesizes completed pilot evidence and firsthand experience to issue retain, adapt-and-rerun, or retire dispositions.
Nearest alternative: Decision, Gate & Allocation — A selective-retention decision follows, but it is explicitly grounded in review of already captured evidence.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Public Administration & Policy
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Using pilot evidence to retain, adapt, or retire a public intervention is policy evaluation.
Related originating lineages:
- Military & Strategic Studies — Military practice supplies the after-action review form.
- Organizational & Management Science — Organizational management contributes continuous improvement and ownership of follow-up.
- Statistics & Experimental Design — Evaluation evidence supplies credible comparison among pilot variations.
Review resolution: Light authoritative-source research resolves the primary-origin disagreement in favor of public administration policy. UK Cabinet Office: Trying It Out—The Role of Pilots in Policy directly documents the defining practice or theory described in the selected origin rationale. Other domains are retained only where the blind reviews identify material co-development or translation; broad application is recorded separately as domain_reach=multi_domain, while origin_mode=cross_disciplinary_synthesis describes the relationship among origin lineages.
Attribution caveat: The boundary with organizational management is substantive because that tradition materially developed or translated part of the mechanism; the cited provenance places the defining form in public administration policy.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
Notes¶
The nearest twin is Feature-Flag Graduation Review. The clean line: this review makes the merits retain / adapt / retire decision on a completed field pilot and records the learning behind it, whereas the graduation review governs the safe flip of a still-live software feature — reversible flag, compatibility, sign-off — taking the merits verdict as given. One judges worth and captures learning; the other governs a reversible release event.
[n1] The after-action review originated in U.S. Army training as a disciplined debrief structured around a few questions — what was supposed to happen, what actually happened, why the difference, and what to sustain or improve. Its lasting contribution is the idea that an event's value is only realized when it is converted into transferable, recorded lessons. ↩
[n2] Hindsight bias — the tendency, once an outcome is known, to see it as having been predictable all along, which makes a retrospective prone to constructing a tidy story that confirms the decision-maker's prior preference. It is why sound reviews fix their evaluation criteria before examining results rather than after. ↩