Circuit Breaker and Cooldown¶
Coordination protocol — instantiates Progress-Guarded Livelock Disruption
Counts repeated failed or non-progressing attempts, trips 'open' to stop the futile retries for a cooldown, then probes cautiously through a half-open state before resuming.
Some livelocks are fed by trying harder: each actor responds to failure by retrying, and the retries themselves keep the system pinned in the stuck region. Circuit Breaker and Cooldown breaks such a loop by stopping rather than re-ordering or arbitrating. It counts failed or non-progressing attempts against a budget; once they cross it, the breaker trips open, forcing a quiescent cooldown that sheds the futile load; after the cooldown it re-enters through a half-open probe that either restores normal operation or trips again. Its signature is that triad — a trip threshold, a hold, and a guarded re-entry — and its defining move is that the cure is temporary inaction.
Example¶
A checkout service calls a payment gateway that starts to flap. Every call times out, the client retries, the retries pile onto the already-struggling gateway, and the two systems thrash — the checkout service is maxed out, unmistakably "doing work," yet no payment completes: a retry-storm livelock. A circuit breaker wraps the call. Once failures cross a budget within a window, it trips open and fails fast for a cooldown of, say, ≈30 s, giving the gateway room to recover and starving the storm of fuel. After the cooldown the breaker goes half-open and lets a single trial call through: succeed, and it closes and traffic resumes; fail, and the cooldown restarts, often longer. Progress is guaranteed not by pushing harder but by deliberately not pushing for a while.
How it works¶
- Budget the failures. A threshold of failed or non-progressing attempts (a count, or a rate over a window) is what trips the breaker — the loop is bounded, not endured.
- Open into quiescence. The open state is a hold that sheds the futile load: calls fail fast, the retries stop, and the coupled system is allowed to settle.
- Re-enter through a checkpoint. The half-open state admits a single or limited probe that governs safe resumption — the recovery checkpoint.
- Close or re-trip. A successful probe closes the breaker; a failed one re-opens it, typically with a longer cooldown.
Tuning parameters¶
- Trip threshold / budget — how many failures, or what rate, before opening. Low trips early (safe, but false trips deny a healthy dependency); high is tolerant but slow to protect.
- Cooldown duration — the length of the open hold; too short re-storms, too long over-denies. Often lengthened exponentially on repeat trips.
- Half-open probe volume — how many trial calls during re-entry. More re-tests faster but risks re-storming a still-fragile dependency.
- Failure definition — what counts toward the budget: errors, timeouts, or flat progress. Mis-defining it trips on the wrong signal.
- Scope / granularity — per-dependency, per-endpoint, or per-shard. Finer scope isolates blast radius; coarser is simpler to operate.
When it helps, and when it misleads¶
Its strength is that it is the fastest way to kill a retry-storm or thrash livelock — stop feeding it — and the half-open probe restores service automatically, without waiting for a human. It also contains blast radius by isolating the failing path from everything upstream. It is the well-worn Circuit Breaker stability pattern.[n1]
Its failure modes: a mis-tuned threshold either trips constantly (denying a healthy dependency) or never (letting the storm run); the cooldown is a blunt hold that also blocks the requests that would have succeeded; and an open breaker masks the underlying fault, so teams sometimes tune the breaker instead of fixing what it is protecting them from. The classic misuse is cranking the threshold up after an incident so the breaker "stops tripping" — silencing the very signal it exists to raise. The discipline is to alarm on trips (a trip is information), match the cooldown to the dependency's real recovery time, and fix the root cause the breaker is buying you time on.
How it implements the components¶
Circuit Breaker and Cooldown fills the stop-and-recover side of the archetype:
retry_budget— the count of failed or non-progressing attempts that trips the breaker and caps the futile retries.quiescence_or_hold_window— the open/cooldown state that halts the storm and lets the coupled system settle.recovery_checkpoint_and_reentry_rule— the half-open probe that governs safe re-entry before full resumption.
It breaks no symmetry between peers (symmetry_breaking_rule — Bounded Priority Rotation / Leader Election or Token Passing), desynchronizes no retries (desynchronization_backoff_policy — Exponential Backoff with Jitter), and detects no cycle itself (non_progress_cycle_detector — Joint-State Cycle Trace).
Related¶
- Instantiates: Progress-Guarded Livelock Disruption — it is the coordination change that replaces a futile retry loop with a deliberate hold and a guarded resumption.
- Sibling mechanisms: Quiescence Barrier · Exponential Backoff with Jitter · Bounded Priority Rotation · Joint-State Cycle Trace · Liveness Watchdog
Editorial Notes¶
Form Classification¶
Form family: Control, Automation & Runtime
Rationale: Counts repeated failed or non-progressing attempts, trips 'open' to stop the futile retries for a cooldown, then probes cautiously through a half-open state before resuming, making its operative form a live operational control that automatically routes, enforces, adapts, or responds during execution.
Independent corroboration: The frozen evidence defines Circuit Breaker and Cooldown as 'Counts repeated failed or non-progressing attempts, trips 'open' to stop the futile retries for a cooldown, then probes cautiously through a half-open state before resuming', so its operative form is Control, Automation & Runtime.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Single lineage
Present-day reach: Specialized
Rationale: Software reliability practice established the closed-open-half-open circuit-breaker pattern with failure budgets, cooldown, and guarded probes.
Related originating lineages:
- Engineering & Design — Electrical protection supplied the breaker metaphor and the physical precedent of opening a circuit to contain a fault.
Review resolution: The closed-open-half-open cooldown pattern cohered as a software-resilience method. Electrical protection supplied the name and fault-interruption precedent, but not the retry budget or guarded half-open probe, so the software lineage remains single and specialized.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
A circuit breaker's cooldown is reactive — it trips on observed failure — which distinguishes it from a planned Quiescence Barrier that holds the whole system at a scheduled synchronization point. Reach for the breaker when failure is the trigger and you want fast, automatic recovery; reach for a barrier when you want everyone to pause together by design, whether or not anything has failed yet.
Draft — generated mechanism page.
[n1] The Circuit Breaker stability pattern — trip on repeated failure, fail fast during a cooldown, then probe through a half-open state before closing — was popularized by Michael Nygard in Release It!. The name borrows from the electrical breaker that opens a circuit to prevent a fault from cascading. ↩