Stop-Rule Postmortem¶
Retrospective review — instantiates Sequential Stopping Boundary Design
Reviews a completed stopping decision after the fact to judge whether the boundary caused avoidable regret or bias, and recalibrates it for the next sequence.
A Stop-Rule Postmortem looks backward at a stopping decision that has already been made and asks whether the boundary that produced it was well-drawn. Its defining trait — the one that separates it from every forward-looking sibling — is that it never decides when to stop; it grades a decision already taken, on the process rather than the luck of the result. It reconstructs the recorded reasoning behind the stop, tests it against what was knowable at the time and what has since come to light, and asks a specific set of questions: did we stop too early and forfeit something better, too late and burn the window, in a way biased against certain candidates or signals? Whatever it finds does not change the past decision; it feeds a recalibration of the boundary — and, where a rule turns out to be systematically wrong or unsafe, an explicit path to suspend or rebuild it — so the next sequence is governed better.
Example¶
A startup accelerator runs cohorts twice a year and, each round, must stop reviewing applications and lock the admitted class by a fixed date. After one cohort, the team runs a postmortem on those stop decisions. They pull the decision record for each round — the thresholds that were in force, the reasons written down for stopping when they did, and which applicants were declined at the margin. The review surfaces a pattern: in the last two rounds the class was locked with a week still on the clock, and the strongest late applicant each time — flagged in the notes but declined because the class was already "full enough" — went on to raise a round elsewhere within the year.
The postmortem does not un-admit anyone or reverse a past call. Its output is a diagnosis and a change. The diagnosis is a premature-commitment bias against late arrivals, driven by a stop rule that let a comfortably-filled class end review before the deadline. The change is a recalibration: hold at least two review slots open until the final week, and route any late applicant who beats the current marginal admit to an exception review rather than an automatic decline. The next cohort runs under the revised boundary — the point of the whole exercise.
How it works¶
- Reconstruct the decision as recorded. Recover the thresholds in force, the written rationale, and the marginal options at the moment of the stop — the trail the decision left, not a memory of it.
- Grade the process, not the outcome. Ask whether the stop was justified by what was knowable then, separating a well-made decision that happened to turn out badly from a genuinely flawed rule.
- Classify the failure, if any. Sort what went wrong into early-stop regret, late-stop cost, or biased/exhaustion-driven stopping — the archetype's named failure modes.
- Recalibrate or flag for override. Feed the finding into a concrete boundary change for next time, and where the rule was systematically unsafe or unfair, invoke the path to suspend or rebuild it rather than merely tune it.
Tuning parameters¶
- Review scope — whether the postmortem examines a single dramatic stop or a whole season of them. A batch review reveals systematic bias a single-case look would miss, but costs more and can wash out an instructive one-off.
- Process-vs-outcome weighting — how much the grade rests on how the decision turned out versus how it was made. Leaning on outcomes catches real miscalibration but courts hindsight; leaning on process is fairer but can excuse a rule that keeps producing bad ends.
- Recalibration aggressiveness — how sharply findings are allowed to move the boundary. Aggressive updates adapt fast but overfit to the last sequence's noise; conservative ones are stable but slow to correct a real flaw.
- Override threshold — how strong the evidence of harm or unfairness must be before the rule is suspended rather than tuned. A low bar protects against unsafe rules but destabilizes the policy; a high bar keeps stability at the risk of running a broken rule too long.
When it helps, and when it misleads¶
Its strength is that it closes the loop the rest of the family leaves open: without a postmortem, a stopping boundary never learns, and the same premature or biased stop recurs cohort after cohort. Done well, it is the guard against turning uncertainty into either chronic over-searching or chronic early-cutting, because it is the only mechanism that checks whether the boundary itself, not just a single decision, is calibrated.
Its central hazard is judging the decision by its result instead of its reasoning — outcome bias, the documented tendency to rate a decision as good or bad according to how it turned out rather than how sound it was given what was known.[n1] A postmortem that indulges it will "correct" a perfectly good rule after one unlucky sequence and leave a genuinely bad rule untouched after one lucky one. Its inverse failure is hindsight-driven over-tuning: chasing the last outcome's noise into the boundary until the rule overfits history. The guarding discipline is to grade the process against what was knowable at decision time, to require a pattern across sequences before moving the boundary, and to reserve the override path for evidence of systematic harm rather than a single regret.
How it implements the components¶
decision_rationale_trace— the postmortem is built on the recorded reasoning behind the stop: it reconstructs and interrogates that trail, which is why a boundary that leaves no rationale record cannot be reviewed at all.override_and_recalibration_path— its whole output is a change to the rule: a recalibration of the boundary for the next sequence, or, where the rule proved systematically unsafe or unfair, an explicit route to suspend or rebuild it.
It sets no live boundary and weighs no efficacy-versus-harm asymmetry in real time — drawing and firing the stopping_boundary from the early_late_error_tradeoff as data accrue is its forward-looking twin, the Sequential Monitoring Stop Rule; this postmortem only grades that boundary after the fact and tunes it for next time.
Related¶
- Instantiates: Sequential Stopping Boundary Design — the retrospective, learning half of the archetype: it audits a boundary and recalibrates it rather than deciding when to stop.
- Consumes: the recorded rationale left by boundary-setting siblings such as the Reservation Value Table and Sequential Monitoring Stop Rule — without their trail, there is nothing to review.
- Sibling mechanisms: Sequential Monitoring Stop Rule · Bayesian Value-of-Information Update · Bid Acceptance Cutoff · Real-Option Exercise Boundary · Reservation Value Table · Secretary-Problem Sampling Rule
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Stop-Rule Postmortem operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it reviews a completed stopping decision after the fact to judge whether the boundary caused avoidable regret or bias, and recalibrates it for the next sequence.
Independent corroboration: The frozen evidence defines Stop-Rule Postmortem as 'Reviews a completed stopping decision after the fact to judge whether the boundary caused avoidable regret or bias, and recalibrates it for the next sequence', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Universal
Rationale: Retrospective calibration of sequential stopping boundaries is statistical review.
Related originating lineages:
- Data Science & Analytics — Data science, analytics, and operational monitoring supplies a parallel or contributing lineage for the mechanism's defining operation: reviews a completed stopping decision after the fact to judge whether the boundary caused avoidable regret or bias, and recalibrates it for the next sequence.
- Mathematics — Mathematical modeling, proof, and abstract-structure practice supplies a parallel or contributing lineage for the mechanism's defining operation: reviews a completed stopping decision after the fact to judge whether the boundary caused avoidable regret or bias, and recalibrates it for the next sequence.
- Medicine & Healthcare — Trial stopping creates regret and bias risks.
- Organizational & Management Science — Postmortems improve governance.
Review resolution: The blind reviewers agree that statistics_experimental_design is the primary origin and differ only on alternate origin disagreement, origin mode disagreement. I preserve every independently explained alternate from both records rather than imposing a numeric cap. I retain cross_disciplinary_synthesis because the combined evidence shows material contributions from several lineages. The broader reach of universal records portability separately from historical provenance; encyclopedia_synthesis=true preserves the affirmative synthesis judgment where either reviewer identified one.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; medium confidence.
Notes¶
[n1] Outcome bias — the tendency to judge the quality of a decision by how it turned out rather than by whether it was sound given the information available when it was made. A stop-rule postmortem must resist it deliberately, grading the boundary on the reasoning that was knowable at the time, so that good rules are not discarded after unlucky sequences and bad rules are not spared after lucky ones. ↩