Rollback Runbook¶
Operational rollback procedure — instantiates Return-Path Design
A rehearsed, pre-authorized procedure for returning a system to a known-good prior state the moment a change goes wrong — fired by explicit trigger criteria and confirmed by explicit recovery checks.
A Rollback Runbook is the pre-written, rehearsed, pre-authorized sequence that returns a running system from a just-changed state that is misbehaving to the previous known-good state. It is the reverse path for an operational change — a deploy, a config push, a schema switch — designed and drilled before the change ships, so it can be executed fast and correctly by whoever is on call, under pressure, without debate. Its defining move is that the reversal is decided in advance: the trigger that fires it, the exact steps, and the definition of "recovered" are all written down and tested, so the moment the change turns bad the team runs a script instead of inventing one. That is what separates it from a user-facing undo, which reverses a single action before commit — this restores whole-system state after a change is already live.
Example¶
A bank ships a new release of its mobile-banking backend at 02:00 behind a blue-green switch. Within minutes the canary's login error rate climbs past the runbook's trigger — an SLO breach that starts burning the night's error budget. The on-call engineer opens the runbook and executes it exactly: shift live traffic back to the previous (blue) version, confirm the health checks go green, and run the reconciliation step that checks whether any transactions written during the ≈9-minute bad window were left half-applied, re-queuing the few that were. Only when every item on the runbook's "recovered" checklist passes is the incident declared over. Because the sequence was drilled in a game day the month before, the whole reversal takes minutes rather than the frantic hour it would take to improvise one live.
How it works¶
- Pre-authorized — no mid-incident debate. The decision to roll back on a given trigger is agreed and signed off before the change ships, so on-call acts without hunting for approval.
- Explicit trigger, not vibes. A specific, measurable condition (SLO breach, error-budget burn, failed health check) fires it — the same condition every time.
- Rehearsed, so it doesn't rot. The runbook is drilled so its commands still work; an untested runbook is a liability discovered at the worst possible moment.
- State reconciliation, not just traffic-shifting. It accounts for whatever the bad version wrote, so the return doesn't leave orphaned or half-applied state behind.
- A written "recovered" checklist decides when the incident is actually over, rather than a hopeful glance at a dashboard.
Tuning parameters¶
- Trigger sensitivity — how fast and how automatically the rollback fires. Tight, automated triggers catch failures in seconds but risk needless rollbacks and flapping between versions; loose triggers wait for human judgment but bleed error budget.
- Automation depth — one-click (or fully automatic) versus a manual step list. More automation is faster and less error-prone under stress but harder to adapt when the failure is weird.
- State-reconciliation depth — how much of what the bad version wrote you reconcile versus discard. Deeper reconciliation protects data integrity but slows the reversal.
- Rehearsal cadence — how often the runbook is drilled; more drills keep it trustworthy but cost engineering time.
- Roll-back vs. roll-forward boundary — the pre-agreed line past which the runbook says don't revert and instead fix forward, because reverting has become the riskier move.
When it helps, and when it misleads¶
Its strength is converting the panic of a bad change into a calm, fast, correct script — the difference between a five-minute blip and an hour-long outage, and between a clean reversal and a data-integrity mess. Its honest limits are real. Not everything is reversible: once a destructive, one-way migration has run there is a point of no return past which "rolling back" silently corrupts data, and the mature move is often to roll forward — fix on the current version — instead.[n1] A runbook also rots as commands and dependencies drift, and an un-rehearsed one fails exactly when needed. The classic misuse is reflexively rolling back to make a symptom disappear without diagnosing it, which can re-bury a latent bug (sometimes one the old version shares too) and destroy the evidence needed to fix it. The discipline that guards against all three is to rehearse the runbook, mark the point of no return explicitly inside the change itself, and require a quick "is this actually a rollback situation?" check before executing.
How it implements the components¶
state_repair_model— returning the system to the prior known-good state, and reconciling anything the bad version wrote, is the state-repair model this mechanism owns.return_trigger_rule— the explicit, measurable condition (SLO breach, error-budget burn, failed health check) that fires the rollback.inverse_success_criteria— the written "recovered" checklist that defines a successful return: old version healthy, checks green, no orphaned state.
It restores system state on a trigger; it does not decide who holds authority to reverse a broader action or hand ownership back (that's De-Escalation Pathway), reverse money or entitlement (Refund or Reversal Protocol), or catch and replay individual failed messages (Dead-Letter Queue and Replay).
Related¶
- Instantiates: Return-Path Design — it is the pre-planned reverse path for an operational state change.
- Sibling mechanisms: Dead-Letter Queue and Replay · Undo or Cancel Flow · Appeal or Review Process · De-Escalation Pathway · Refund or Reversal Protocol · Resubmission with Preserved State · Return Authorization Workflow · Return-Reason Dashboard · Reverse Logistics Channel · Round-Trip Journey Test · Unsubscribe or Exit Path
Editorial Notes¶
Form Classification¶
Form family: Protocol, Workflow & Routine
Rationale: Rollback Runbook operates as a repeatable ordered procedure or handoff sequence that coordinates action because it a rehearsed, pre-authorized procedure for returning a system to a known-good prior state the moment a change goes wrong — fired by explicit trigger criteria and confirmed by explicit recovery checks.
Independent corroboration: The frozen evidence defines Rollback Runbook as 'A rehearsed, pre-authorized procedure for returning a system to a known-good prior state the moment a change goes wrong — fired by explicit trigger criteria and confirmed by explicit recovery checks', so its operative form is Protocol, Workflow & Routine.
Nearest alternative: Control, Automation & Runtime — Rollback Runbook includes features of a live operational control that automatically routes, enforces, adapts, or responds during execution, but its defining operation is a repeatable ordered procedure or handoff sequence that coordinates action.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Convergent development
Present-day reach: Multi-domain
Rationale: Preauthorized, rehearsed restoration to a known-good state is a canonical software-operations runbook.
Related originating lineages:
- Engineering & Design — Safety and maintenance procedure traditions materially contribute triggers and recovery verification.
Review resolution: Both blind reviewers agree that computer_science is the primary historical origin. Explicit reconciliation of origin mode disagreement starts from reviewer_a’s mechanism-specific evidence: Preauthorized, rehearsed restoration to a known-good state is a canonical software-operations runbook. Reviewer A proposed alternates=engineering_design, origin_mode=convergent, domain_reach=multi_domain, and encyclopedia_synthesis=false; reviewer B proposed alternates=engineering_design, origin_mode=single_lineage, domain_reach=multi_domain, and encyclopedia_synthesis=false. The final record retains every independently supported alternate from either review (engineering_design) without an arbitrary cap, selects origin_mode=convergent to represent the combined lineage evidence, and keeps domain_reach=multi_domain and encyclopedia_synthesis=false from the more mechanism-specific assessment. Present-day transfer is recorded as reach and is not treated as proof of historical origin.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
A Rollback Runbook is only as valid as the reversibility of the change it guards. The decisive check lives not in the runbook but in the change itself: has anything irreversible — a destructive migration, an external side effect already sent — happened yet? Keep a live "still rollback-able?" marker in the deploy and switch to forward-recovery the moment it flips.
[n1] Roll-forward (forward recovery) — recovering by moving to a new corrected state rather than reverting to a prior one. In fault-tolerant systems the two are complementary: backward recovery (rollback) is cheap and safe until an irreversible step makes forward recovery the only sound option. ↩