Skip to content

Half-Open Circuit Probe

Circuit-breaker probe — instantiates Synchronized Release Dampening

After a tripped circuit blocks all traffic to a failed dependency, it lets a single trial request through at intervals and reopens the floodgates only once a probe succeeds — so recovery is tested by one caller, not the whole herd.

A Half-Open Circuit Probe is the recovery-detecting move inside a circuit breaker. When a downstream dependency has failed and the breaker is open — fast-failing every call so the crowd stops piling onto a resource that cannot answer — the probe periodically shifts the breaker to half-open and admits exactly one (or a few) trial requests. If the probe succeeds, the breaker closes and normal traffic resumes; if it fails, the breaker snaps open again and the herd stays off. Its defining move is that the recovery test is a single probe, not the whole waiting population rediscovering health at the same instant. It produces the "capacity is back" signal safely, by making one caller — not ten thousand — pay the cost of asking too early.

Example

An e-commerce checkout calls an inventory-reservation service to hold stock. Under a traffic spike the inventory service starts timing out, and a circuit breaker opens so checkout stops waiting on it and instead falls back to a "reserve shortly" path — which keeps checkout alive and, crucially, takes the load off the struggling service. Every ~10 seconds the breaker goes half-open and lets a single reservation call through as a probe. While that probe keeps failing, the breaker stays open and the ten thousand in-flight checkouts stay off the service's back. When a probe finally succeeds, the breaker closes and reservations resume — but the key property is what did not happen: the moment the inventory service flickered back to life, it was greeted by one trial request rather than the entire backlog testing it simultaneously and knocking it straight back down.

How it works

  • Three states. Closed (traffic flows), open (all calls fast-fail), half-open (a trial trickle is allowed).
  • Probe on a timer. After the breaker trips, a timer — often itself backed off — moves it to half-open and admits one or a few trial calls.
  • Gate the reopen on the probe result. Only a successful probe closes the breaker; a failure returns it to open. The gate's open/closed decision is driven by an active health test, and only the probe bears the risk of being wrong.
  • Optionally ramp, don't slam. Reopen gradually (half-open → limited → full) rather than jumping to full traffic on a single success.

Tuning parameters

  • Probe interval — how often, and with what backoff, half-open trials are attempted; longer intervals spare a sick service but extend the outage.
  • Trial count and success threshold — how many probes must succeed before the breaker is declared closed; more probes resist false "healthy" readings but delay recovery.
  • Reopen ramp — binary reopen versus gradual widening; a binary reopen answers "can you serve one?" and then hands over the whole herd.
  • Trip threshold — how many failures open the breaker in the first place, trading sensitivity against flapping.

When it helps, and when it misleads

Its strength is that it prevents recovery from becoming a second outage and contains the blast radius of a flapping dependency: the probe, not the crowd, absorbs the risk of testing too soon.[1] Its signature failure is oscillation — a service that is only barely back can pass a single probe, get slammed by the full returning load, die again, and re-open the breaker, thrashing indefinitely when the reopen is binary rather than ramped. Too-long probe intervals are the quieter failure: a healthy service kept locked out extends its own outage. The classic misuse is reopening straight to 100% traffic on one success — the probe verified the service could handle one request, not the herd. The discipline is to ramp re-entry, require several successes, and hand the reopened flow to a capacity-aware admission stage so a green probe does not immediately summon the whole crowd.

How it implements the components

  • capacity_recovery_signal — the probe result is the recovery signal: a successful trial is the health evidence that gates re-entry, produced without exposing the resource to the full herd.
  • finite_choke_point — it exists to shield one specific failed dependency, keeping the breaker's protection scoped to the resource that actually fell over.
  • admission_gate — the open / half-open / closed states are an admission gate over who reaches the choke, from "no one" to "one trial" to "everyone."

It decides when capacity is back but does not meter the returning herd at that capacity — that is Capacity-Aware Reconnect Queue, the natural consumer of its signal; nor does it decorrelate retries (Exponential Backoff with Jitter) or partition the population into waves (Cohort-Based Reactivation).

References

[1] The circuit breaker, with its closed / open / half-open states, is a widely used stability pattern popularized by Michael Nygard's Release It!. It is named here as a real, established pattern; the half-open probe is the specific part of it that turns "is the dependency healthy yet?" from a question the whole herd asks at once into one a single trial request asks on the herd's behalf. ↩