Incident Recovery Plan¶
Plan / runbook — instantiates Recovery Trajectory Management
A bounded plan for returning one disrupted system to service — it records the blast radius, phases the recovery, and gates each reentry so the fix does not trigger a second failure.
An Incident Recovery Plan governs the return of a single bounded system to service after a specific incident. Its endpoint is not in question — the system should work again — so all of its attention goes to the path: what exactly was displaced, which phases the recovery must move through, and what condition must be met before each gate opens. Its defining move is to treat post-incident recovery as phased and gated rather than a single "bring it back up." A damaged system brought back all at once tends to fail again — a cold cache stampede, a thundering herd, a dependency that is up but not yet consistent — so the plan sequences reentry deliberately and refuses to advance a phase merely because time has passed or pressure is high.
Example¶
A payments platform suffers a cascading outage: a region's primary datastore fails over badly, and half the API fleet is now serving errors while queues back up. The incident recovery plan governs the climb out. It opens with a displacement record — which services are down, which are degraded, the blast radius across dependent merchants, and what is still unknown (is the failover replica consistent?). It lays out a phase map: stabilize the datastore, restore read traffic, restore writes, drain the backlog, then reopen full merchant traffic. Between each phase sits a gate: writes do not resume until replica lag is under a set bound and integrity checks pass; full traffic does not return until error rates hold flat under a staged 10% → 50% → 100% reentry.
The payoff is that the plan stops the two classic post-incident mistakes: flipping everything back at once (which re-overloads a fragile system) and declaring victory when the dashboard goes green but the write path is still lossy. Each gate is a small, checkable promise — "this must be true before we proceed" — so the recovery is auditable in real time and reversible if a gate fails.
How it works¶
- Record the displacement first. Before touching the system, the plan captures what moved out of working function — down and degraded services, dependency health, affected external parties, and the open unknowns — so recovery starts from observed state, not assumption.
- Phase the climb. It decomposes the return into ordered phases whose risks differ (stabilize → restore reads → restore writes → drain backlog → full reentry), because a system responds differently at each stage.
- Gate every transition. Each phase boundary carries an explicit entry condition — a metric bound, an integrity check, a soak period — and a controlled, ramped reentry rather than a full cutover, with a defined fallback if the gate fails.
- Refuse time-based advancement. A phase changes on observed readiness, never because an SLA clock or an anxious stakeholder is pushing.
Tuning parameters¶
- Phase granularity — a few coarse phases versus many fine ones. Finer phasing catches trouble earlier but lengthens the recovery and adds coordination overhead.
- Gate strictness — how conservative each entry condition is. Tight gates prevent relapse but slow the return and can trap the system in a phase during a genuine emergency.
- Reentry ramp shape — how fast traffic or load is reintroduced (e.g., 1% → 10% → 100% versus a single cutover). Gentler ramps surface latent faults safely at the cost of time.
- Fallback depth — whether a failed gate pauses, rolls back one phase, or aborts to the stabilized floor. Deeper fallback is safer but more disruptive.
- Record fidelity — how much of the blast radius and unknowns the displacement record captures up front versus discovering mid-recovery.
When it helps, and when it misleads¶
Its strength is that it makes a chaotic bring-up legible and reversible: every step is a checkable condition, so the team can prove where they are and back out cleanly. It directly counters premature reentry — the plan's whole architecture is built to stop "it's up" from being mistaken for "it's well."
Its failure mode is that a plan is only as honest as its blast radius estimate.[n1] If the displacement record misses a silently corrupted dependency or an affected downstream, the phases and gates are governing the wrong system and can wave through a recovery that is validated on paper and broken in fact. The classic misuse is gates that check activity ("failover completed") instead of function ("writes are durable and consistent"). The guarding discipline is to write each gate as a function-level condition tied to the recorded blast radius, and to keep re-scoping that record as the incident reveals more than was first visible.
How it implements the components¶
disruption_and_displacement_record— the opening artifact: down and degraded services, dependency health, external blast radius, and open unknowns, captured as observed state.phase_map— the ordered set of recovery phases (stabilize → reads → writes → backlog → full reentry), each with its own risk profile.phase_gate_and_reentry_rule— the entry condition and ramped, reversible reentry that governs each phase boundary.
It does not choose or justify the endpoint, weigh who bears the burden, or coordinate outside aid — recovery_endpoint_definition, equity_and_burden_monitor, and external_support_interface belong to Community Recovery Plan, its nearest twin. The line between them is sharp: an incident recovery plan takes "restore the service" as given and governs one bounded system's technical path back, while a community recovery plan decides what "recovered" should even mean for a whole population and who pays for it.
Related¶
- Instantiates: Recovery Trajectory Management — this plan supplies the recorded-and-gated path a single system takes back to service.
- Consumes: Stabilization Checklist confirms the floor that the plan's first phase must reach before restoration proceeds.
- Sibling mechanisms: Community Recovery Plan · Service Restoration Runbook · Critical Function Triage Matrix · Phased Restoration Schedule · Recovery Dashboard
Editorial Notes¶
Form Classification¶
Form family: Representation, Specification & Plan
Rationale: Incident Recovery Plan operates as a non-executable information artifact that externalizes static or prospective structure because it a bounded plan for returning one disrupted system to service — it records the blast radius, phases the recovery, and gates each reentry so the fix does not trigger a second failure
Independent corroboration: The frozen evidence defines Incident Recovery Plan as 'A bounded plan for returning one disrupted system to service — it records the blast radius, phases the recovery, and gates each reentry so the fix does not trigger a second failure', so its operative form is Representation, Specification & Plan.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Blast-radius-aware, gated restoration of a disrupted service is established site-reliability and disaster-recovery practice.
Related originating lineages:
- Disaster Management & Risk Reduction — Phased recovery and reentry planning independently developed for disrupted communities and operations.
- Engineering & Design — Commissioning and safe restart procedures materially shape evidence-gated return to service.
Review resolution: Both reviewers independently assign computer_science as the primary originating domain, so that shared primary is retained. Alternate domains are the union of reviewer-identified formative or independently originating lineages; later application settings alone are excluded. The final form materially composes methods or concepts from more than one formative domain. It has established independent use across several domains, but that does not make it domain-free. The encyclopedia entry makes that composition explicit.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] In site-reliability practice, blast radius names the full extent of what an incident (or a recovery action) can affect. Sizing it accurately is the precondition for scoping recovery correctly; an underestimated blast radius is why a "completed" recovery can still be sitting on undiscovered damage. ↩