Skip to content

Host-Dependency Fallback Drill

Failure-rehearsal drill — instantiates Layer-Appropriate Capability Placement

Rehearses host failure — degraded, disconnected, incompatible, or withdrawn — before the dependency is load-bearing, so the subsystem's graceful-degradation rules are proven rather than assumed.

A Host-Dependency Fallback Drill deliberately breaks the subsystem's connection to the host — under controlled conditions, before real users depend on it — to find out what actually happens when the host degrades, disconnects, changes incompatibly, or goes away entirely. A layer built on a richer host quietly accumulates assumptions that the host is always present, fast, and unchanged; the drill turns those assumptions into tested facts by injecting the failure and watching whether the subsystem degrades gracefully, keeps its offline obligations, and stays within a sane dependency budget — or falls over. Its defining move is that it validates resilience by rehearsal rather than by inspection: it doesn't read the design and reason about failure, it causes the failure and measures the response.

Example

A retail chain's point-of-sale terminals lean on a cloud service for live pricing, promotions, and card authorization. On paper there's a story for "the internet goes down." The Fallback Drill tests it for real: in a pilot store, after hours, the team blocks the terminals' path to the cloud and rings up a normal transaction. The first drill is ugly. The terminal can still sell from a cached price book (graceful degradation works) — but card auth hangs for ninety-odd seconds before timing out instead of falling back to offline-approval-with-later-settlement, and a lane freezes entirely when the loyalty lookup gets no answer. Those are exactly the behaviours a design review had rated "handled."

The drill also exposes a creeping dependency-budget problem: three separate features now block on the same cloud call, so one outage stalls all three at once. The fixes — a bounded auth timeout with an explicit offline-approval rule, and a loyalty lookup that fails open — are written and then re-drilled until the store can trade through a full disconnection.

How it works

  • Pick the failure mode, then cause it. Degraded (slow/partial), disconnected (offline), incompatible (host changed its contract), or withdrawn (host gone) — each is injected deliberately, not argued about on a whiteboard.
  • Run the real workflow against it. Exercise the subsystem's actual user paths while the host is impaired, and observe behaviour end to end rather than component health.
  • Score against the degradation contract. Check the response against what the subsystem promised: does it degrade gracefully, honour its offline-operation requirement, and stay within its host-dependency budget?
  • Fix and re-drill. Failures become concrete fixes to the fallback rules; the drill repeats until the impaired-host scenario is survivable, then recurs on a schedule so drift is caught.

Tuning parameters

  • Blast radius — a single feature in a sandbox versus the whole subsystem in a production-like environment. Wider is more honest and more dangerous.
  • Failure realism — a clean disconnect versus a slow, flapping, half-broken host. Messy failures are the ones that actually happen and the ones designs miss.
  • Notice — an announced game-day versus a surprise injection. Surprise tests the humans and the runbooks too; announced tests only the system.
  • Cadence — one-off before launch versus a recurring drill. Dependencies drift, so a passing drill decays; recurrence is what keeps it true.
  • Recovery scope — whether the drill also rehearses coming back — re-sync, backfill, reconciling state written while offline — not just surviving the outage.

When it helps, and when it misleads

Its strength is that it converts "we handle that" from a claim into evidence, and it surfaces the specific, unglamorous gaps — the unbounded timeout, the fail-closed lookup, three features silently sharing one dependency — that only appear when the host is genuinely missing. It is the practical arm of designing for graceful degradation.[1]

Its limits are real too. A drill only tests the failure modes you thought to inject, so it can breed false confidence about the ones you didn't; and a badly scoped drill can cause the very outage it was meant to prevent. The classic misuse is the box-checking drill — run once, filed as "passed," never repeated — so a dependency that has since doubled sails on under a stale certificate of resilience. The discipline that guards against this is to treat every drill as perishable: schedule its recurrence, vary the injected failure, and rehearse recovery, not just survival.

How it implements the components

The drill operationalizes the resilience side of the archetype — the specs that only become real once they have survived an injected failure:

  • fallback_or_graceful_degradation_rule — the drill is where these rules are proven or broken; a rule that has never survived an injected failure is only a hypothesis.
  • offline_operation_requirement — disconnecting the host is the direct test of whether the subsystem meets its stated offline obligations.
  • host_dependency_budget — by revealing how many paths block on the host and how hard, the drill measures actual dependence against the budget the layer is supposed to keep.

It exercises the substitution path when it withdraws the host, but building that stand-in belongs to Temporary Local Shim with Expiry; and it tests, rather than defines, the delegating interface owned by Host-Service API Delegation.

Notes

A drill only has value if the subsystem has a fallback to exercise; run against a system with no graceful-degradation path, it simply reproduces the outage. Pair the first drill with at least a minimal offline mode or a Temporary Local Shim with Expiry to degrade onto — otherwise the drill becomes a demonstration of the problem rather than a test of the cure.

References

[1] Graceful degradation — the design property that when a dependency fails, the system continues at reduced capability rather than failing completely. The drill is how a claimed degradation path is verified; the broader practice of deliberately injecting failures into a running system to test resilience is known as chaos engineering. ↩