Skip to content

Recovery Drill and Restore Test

Recovery drill — instantiates Eventual-Occurrence Containment Design

Actually restores the system from a simulated occurrence, end to end and on the clock, to prove rather than assume that recovery works and critical functions return within their targets.

A recovery plan on paper is a hypothesis. Recovery Drill and Restore Test is the mechanism that turns it into evidence by executing it — restoring from backups, failing over to the standby, rebuilding the environment — under realistic conditions and against the clock, then checking that the critical functions came back, and came back in time. Its defining move is refusing to grade recovery on the existence of a plan; it grades on a live restore that either meets its recovery-time and recovery-point targets or does not. Because the archetype assumes the bad event will occur, "we have backups" is not an answer — "we restored from them last quarter, end to end, inside our data-loss target" is.

Example

A bank runs a quarterly disaster-recovery drill on its core ledger. The stated targets are a four-hour recovery time and a five-minute recovery point. On paper the runbook meets both. The drill restores the ledger from the previous night's backup onto the standby site with the primary deliberately taken dark — and it takes fourteen hours, not four, because the backups restore serially and nobody had timed the full data set at production scale. It also surfaces that a downstream reconciliation service cannot reach the standby's network, so even once the ledger is up, a critical function is still dead.

Neither gap was visible on paper; both are now fixable before a real outage rather than discovered during one. The drill's entire value is the distance between the planned four hours and the measured fourteen — a distance you only ever learn by actually doing the restore.

How it works

  • Simulate a real occurrence — take the primary offline, corrupt a data set, or fail a region — rather than testing components in isolation.
  • Execute the actual recovery-and-recurrence plan end to end: restore, fail over, rebuild, and bring critical functions back in their real dependency order.
  • Measure against targets: did recovery time and recovery point meet their objectives, and did the minimum critical functions return — not just the headline service?
  • Record every gap between planned and observed and feed the fixes back into the plan. The drill's output is a corrected, proven plan, not a pass/fail stamp.

Tuning parameters

  • Realism — tabletop walk-through, isolated restore, or full failover with the primary genuinely dark. More realism finds more real gaps but risks the drill itself causing an incident.
  • Scope — one component, one service, or the whole interdependent system. Whole-system drills catch the cross-dependencies (the unreachable reconciliation service) that component tests never will.
  • Announced vs. surprise — scheduled and staffed, or unannounced. Surprise drills test the humans and the on-call path, not just the machinery, at higher operational risk.
  • Target strictness — how tight the recovery-time and recovery-point objectives are set. Tighter targets expose more gaps but cost more to meet.
  • Cadence — how often the drill runs. Systems drift, so a plan proven a year ago is again just a hypothesis.

When it helps, and when it misleads

Its strength is that it closes the most dangerous gap in this archetype — the difference between believing you can recover and having done it. It routinely surfaces restore times that are multiples of the assumed target, backups that don't actually restore, and critical dependencies nobody mapped, all while the stakes are a drill and not a disaster.

The failure mode is the rehearsed drill that always passes: announced far ahead, run on a clean staging copy, scripted so the known-good path is the only one exercised — a demonstration, not a test, proving only that a happy path works on a good day. Its cousin is drift: a drill that validated last year's architecture and silently no longer matches production. The classic misuse is running the drill to earn a green audit checkmark rather than to find gaps, which quietly selects for easy, unrealistic scenarios. The discipline is to vary and harden the scenario over time, to drill the whole interdependent system rather than the convenient slice, and to treat any drill that "passed" too easily as a sign the scenario was too kind.[1]

How it implements the components

  • recovery_and_recurrence_plan — the drill executes this plan in anger and returns it corrected; a plan that has survived a real restore is the mechanism's product.
  • critical_function_preservation_plan — it verifies that the minimum critical functions actually return, in order and within targets, not merely that the primary service reboots.

It does not detect the occurrence (Sentinel Event Monitoring), design the safe or degraded state it restores from (Degraded-Mode Runbook), or handle the human aftermath of notifying and remedying affected parties and killing the root cause — that is Post-Incident Recurrence Review's.

Notes

Recovery Drill sits deliberately close to two siblings and stays distinct from both. Failure-Injection Test breaks things to see what happens (does the system tolerate the fault?); this drill assumes the failure has already happened and asks whether you can come back from it. And where Post-Incident Recurrence Review learns from a real occurrence after the fact, the drill manufactures a rehearsed one beforehand — proof purchased before the bill comes due.

References

[1] RTO / RPO — Recovery Time Objective (how quickly service must return) and Recovery Point Objective (how much recent data may be lost). A recovery drill exists to test claims against these two targets rather than accept them on paper. ↩