Skip to content

Fault Tolerant Operation

Keep operating despite partial failure by detecting, isolating, masking, bypassing, or compensating for failed components.

The Diagnostic Story

Symptom: One component fails and the whole system goes down. Operators know something broke but can't tell what or where until users are already affected. Local faults leak into shared state, corrupting data or triggering cascades far from the original failure. The backup exists, but nobody is sure what happens operationally when primary fails.

Pivot: Define the critical function separately from the components that implement it, add detection and isolation so faults can be contained, choose a bounded continuation mode that preserves the critical function within explicit limits, and specify recovery or escalation so the continuation does not become indefinite hidden degradation.

Resolution: Local faults no longer automatically cause global functional collapse. The critical function continues within stated bounds during partial failure. Operators have clearer signals and procedures rather than improvising under stress, and shared state remains reconcilable during and after faults.

Reach for this when you hear…

[on-call SRE] “The database replica went down and took the entire checkout flow with it — we need circuit breaking so one unhealthy dependency doesn't cascade into a full site outage.”

[aviation] “If a hydraulic system fails, I need to know right now what the aircraft can and cannot do — not improvise it under pressure at altitude.”

[hospital operations] “We lost the electronic records system but the nurses kept treating patients because we'd rehearsed the downtime procedure — that's fault tolerance working.”

Mechanisms / Implementations

  • Fault Detection and Diagnosis: implements the sensing and classification side of the archetype.
  • Fault Isolation: implements the containment side.
  • Error Correction: implements tolerance by repairing or masking incorrect outputs before they harm the protected function.
  • Redundant Voting: implements tolerance by comparing multiple replicas, sensors, reviewers, or calculations and trusting an acceptable quorum.
  • Bypass Routing: implements tolerance by moving work, traffic, authority, information, or flow around a failed element.
  • Degraded Operation Mode: implements tolerance by preserving the most important function at reduced feature scope, capacity, precision, or automation.
  • Self-Healing Repair Loop: implements tolerance by detecting a fault, applying a bounded repair, verifying recovery, and reintegrating the component.
  • Manual Continuity Workaround: implements tolerance through a rehearsed human alternate procedure when the normal automated path fails.
  • Service Continuity Runbook: implements the archetype as an operational document that coordinates detection, isolation, continuation, escalation, and recovery.

Abstractions this archetype builds on — directly (a source ingredient) or as a related pattern. Links follow the typed catalog namespace.

Built directly on (3)

Also references 10 related abstractions

Variants

Narrower or domain-specific specializations that share this archetype's core structure. Recognized variants are established; candidate variants are provisional.

Error-Masking Operation · mechanism family variant · recognized

Continue operation by detecting local errors and masking, correcting, or outvoting them before they affect the protected function.

Fault Containment Continuation · risk or failure variant · recognized

Continue the protected function by isolating the failed portion and allowing unaffected portions to keep operating.

Bypass Continuation · implementation variant · recognized

Keep the critical function active by routing work, flow, authority, or information around a failed element.

Self-Healing Operation · implementation variant · candidate

Use an internal repair loop to detect faults, apply corrective actions, verify recovery, and continue operation with minimal outside intervention.

Degraded Continuation Mode · risk or failure variant · merge review

Continue only the most important parts of the function when full-quality operation is impossible after a fault.