Skip to content

Token-Bucket Admission Gate

Rate-limiting admission gate — instantiates Synchronized Release Dampening

Admits requests only as fast as tokens refill at a fixed rate, letting a bounded burst through but no more — converting a synchronized surge into a metered stream.

A token bucket holds tokens that refill at a steady rate r, up to a cap B. Each admitted request spends one token; when the bucket is empty, further requests are held or turned away until a token refills. Token-Bucket Admission Gate puts this at the mouth of a choke point so arrivals are admitted no faster than r. Its defining move — and what distinguishes it from the archetype's concurrency cap — is that it meters the arrival rate over time, not the number of actors inside at once, and the burst cap B is a deliberate, bounded tolerance for a small clump, so brief benign bursts pass while a sustained flood is smoothed to r. Its release is clock-driven: tokens accrue with time, so a synchronized surge is paced into a stream flowing at exactly the rate the resource can sustain, with the excess handed back to the caller as "not yet — try again."

Example

A freeway on-ramp has a metering light. At rush hour an arterial dumps a platoon of 40 cars onto the ramp within seconds — a synchronized release that, merged all at once, triggers a merge-collapse and a stop-and-go shockwave on the freeway. Ramp metering is a token bucket: the light releases one car every few seconds (the refill rate r, set to the freeway's spare merge capacity), with a short green-on-arrival tolerance for a tiny cluster (the bucket B).

The platoon still arrives synchronized, but it leaves the ramp decorrelated — one car per token — merging at a rate the mainline can actually absorb. Cars arriving faster than r wait in the ramp queue rather than on the freeway; the queue is where the surge is held, and the freeway sees a smooth trickle. The identical shape guards an API: requests are admitted at r per second with burst B, and callers over the limit get a "retry shortly" instead of admission. In both cases the finite resource downstream is protected not by being bigger, but by never being handed the whole synchronized wave at once.

How it works

  • Tokens refill at rate r, capped at B. r is set to the choke point's sustainable throughput; B bounds how large an instantaneous burst may pass before smoothing engages.
  • Admit by spending a token. Each request consumes one; an empty bucket means the request is rejected or queued with a "come back when a token is available" signal (in HTTP, a 429 with a Retry-After hint).
  • Meter rate, not occupancy. Once admitted, a request isn't tracked — the gate controls arrivals per unit time and is deliberately blind to how many are concurrently in flight.
  • Refill is time-based, so admission keeps flowing through idle-then-bursty patterns, converting the burst into a stream at r with B of slack.

Tuning parameters

  • Refill rate r — the sustained admission rate; set it to the resource's true safe throughput. Too high and the gate is decorative; too low and it throttles healthy load.
  • Bucket size B (burst) — how big a momentary burst passes before smoothing bites. Larger B is friendlier to bursty-but-benign traffic but lets a bigger mini-herd through in one gulp; B = 1 is a strict, burst-free rate.
  • Empty-bucket behavior — reject immediately (shed load, fail fast) vs. queue-and-wait (smoother, but adds latency and a queue you must bound).
  • Retry-After signaling — whether a rejected caller is told when to return. A good, jittered hint desynchronizes retries; a missing one invites a retry storm.
  • Scope / key — one global bucket vs. per-client or per-tenant buckets. Per-key stops one caller from spending everyone's tokens; global protects the shared resource as a whole.

When it helps, and when it misleads

Its strength is precise control of sustained arrival rate while still tolerating harmless micro-bursts — it turns a synchronized surge into a stream at exactly the rate the resource can take, and it composes cheaply with almost anything upstream or down.

Its failures are mostly mis-set dials. Put r above real capacity and the gate is theater: it admits at a rate the choke point can't serve. A generous B is a loaded gun — it lets a genuine herd through in a single gulp before metering ever engages. And a gate that rejects without a good Retry-After merely relocates the herd: every rejected client retries at the same moment and re-synchronizes, which is why the escalating, jittered backoff belongs on the client side. The classic misuse is tuning r to hit a throughput target — running the dial backward from a business number rather than forward from a capacity measurement. The discipline is to set r from measured capacity, keep B no larger than a truly harmless burst, and always pair a rejection with a jittered retry hint so the gate desynchronizes retries instead of breeding them.[1]

How it implements the components

  • admission_gate — it is the gate: the token check is the admit-or-deny decision made at the boundary of the choke point, on every arrival.
  • retry_backoff_contract — an empty bucket produces the "no admission now; return when a token refills" contract (the 429-plus-Retry-After shape), turning a rejection into an orderly later attempt rather than an immediate re-bang.

It does NOT cap *concurrent occupancy — that is Semaphore-Limited Release (arrivals-per-second vs. holders-at-once) — and it does not own the client's backoff curve: Exponential Backoff with Jitter decides how the retry delay escalates and scatters, while this gate only says "not yet, retry after t." It grants no exemptions either — that is Priority Bypass Token.*

Notes

Rate and concurrency are orthogonal, and a token bucket controls only the first. A workload of slow, long-held requests can stay comfortably under the rate limit yet still pile up N-deep at the resource, because each admitted request lingers. When both dimensions matter, a token bucket in front of a Semaphore-Limited Release covers what neither does alone — one caps how fast work arrives, the other how much is in flight at once.

References

[1] The token bucket permits bursts up to B while enforcing an average rate r; the closely related leaky bucket enforces a strict constant output rate with no burst allowance. Choosing between them is really a choice about whether short bursts are acceptable — token bucket says yes up to B, leaky bucket says no. ↩