Idempotency Key & Deduplication Store¶
Deduplication store — instantiates Nested and Distributed Transaction Coordination
Stamps each request with a caller-supplied unique key and remembers the outcome, so a retried or duplicated request produces its effect exactly once.
In any coordinated transaction that retries on failure, the same request can arrive twice — the first attempt succeeded but its acknowledgement was lost, so the caller sends it again. Idempotency Key & Deduplication Store makes the second arrival harmless: the caller attaches a unique idempotency key to the request, and the receiver keeps a durable store mapping each key it has seen to the result it produced. A fresh key is processed and its outcome recorded under that key; a repeat key skips execution entirely and returns the stored result. Its defining idea is a request-level memory: not "was this data written?" but "have I already handled this exact request?" — a keyed lookup that collapses many deliveries of one intent into a single effect, no matter how the retries arrive.
Example¶
A payments API receives a "charge $40" request carrying the idempotency key chk_9f3a1 that the client generated for this one checkout. The service processes it, charges the card, and stores chk_9f3a1 → {charged, receipt #R-77}. The client's network drops the response, so its retry logic re-sends the identical request with the same key. This time the service finds chk_9f3a1 already in the store, does not touch the card again, and replays the stored receipt #R-77. The customer is charged once and sees one receipt, even though the request was delivered twice. A genuinely new purchase carries a new key and is charged normally — the store distinguishes "the same request again" from "a different request that happens to look similar" purely by the key.
How it works¶
- Caller mints the key. The idempotency key is generated once by the client per logical operation and reused across all retries of that operation, so duplicates are self-identifying.
- Check-then-act, atomically. On arrival the receiver reserves the key (rejecting a concurrent second copy), executes only if the key is new, and records the outcome under the key before responding.
- Replay the recorded result. A repeat key returns the stored response rather than re-executing, so the caller gets a consistent answer to every copy of its request.
- Bound the memory. Keys are retained for a defined window — long enough to outlive realistic retry storms — then evicted.
Tuning parameters¶
- Key retention window — how long a processed key is remembered. Longer windows catch late duplicates but grow the store; too short and a delayed retry re-executes because its key has already been forgotten.
- Scope of a key — per-endpoint, per-account, or global uniqueness. Narrow scope keeps keys small and collision-proof within context; global scope guards against cross-path replays at higher storage cost.
- In-flight collision handling — whether a second copy arriving while the first is still executing blocks, rejects, or waits for the result. This dial sets behaviour under concurrent duplicates, the trickiest case.
- Result fidelity — store the full response versus just a "done" marker. Full replay gives byte-identical answers to retries; a marker is cheaper but forces the caller to re-fetch state.
When it helps, and when it misleads¶
Its strength is converting an at-least-once delivery world — the only kind a retrying distributed system can cheaply offer — into effectively exactly-once behaviour at the operation level, which is what makes safe retries, recovery replay, and duplicate messages tolerable at all.[n1]
Its failure mode is key discipline: dedup is only as good as the caller's keys. If the client mints a new key on each retry, the store sees unrelated requests and the duplicate charges anyway; if it reuses a key across genuinely different operations, the store suppresses a real second action. A subtle trap is a retention window shorter than the retry horizon, so a slow duplicate lands after its key has expired. The classic misuse is bolting a dedup store onto an operation whose side effects escape the store — it records "handled" after an external email already went out twice. The guarding discipline is to make the caller own stable keys, keep the record-outcome step inside the same atomic boundary as the effect, and size retention against the longest plausible retry, not the average.
How it implements the components¶
idempotency_and_replay_safeguard— this is its whole purpose: a keyed store that guarantees a repeated or replayed request applies exactly once.failure_timeout_and_partition_model— the retention window and in-flight collision rules define how the store behaves precisely when retries and partitions produce duplicates.
It does not couple a state change to an outbound message atomically (atomicity_and_consistency_objective) — that is the Transactional Outbox/Inbox Pattern, which consumes this store for its inbox-side dedup — and it neither reserves resources nor sequences repairs (compensation_and_reconciliation_plan), which belong to Escrow or Reservation Hold and Manual Reconciliation Workbench.
Related¶
- Instantiates: Nested and Distributed Transaction Coordination — this is the exactly-once safeguard every retry-driven mechanism leans on.
- Sibling mechanisms: Commit-Log Recovery Replay · Transactional Outbox/Inbox Pattern · Saga Orchestration · Saga Choreography · Escrow or Reservation Hold · Manual Reconciliation Workbench · Quorum or Consensus Commit
Editorial Notes¶
Form Classification¶
Form family: Control, Automation & Runtime
Rationale: Idempotency Key & Deduplication Store operates as a live operational control that automatically routes, enforces, adapts, or responds during execution because it stamps each request with a caller-supplied unique key and remembers the outcome, so a retried or duplicated request produces its effect exactly once
Independent corroboration: The frozen evidence defines Idempotency Key & Deduplication Store as 'Stamps each request with a caller-supplied unique key and remembers the outcome, so a retried or duplicated request produces its effect exactly once', so its operative form is Control, Automation & Runtime.
Nearest alternative: Record, Log & Register — The store's primary function is execution-time duplicate suppression and result replay, not historical accountability.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Single lineage
Present-day reach: Specialized
Rationale: Caller-supplied keys plus a deduplication store are a standard distributed API and messaging construction for safe retries.
Review resolution: Both reviewers independently assign computer_science as the primary originating domain, so that shared primary is retained. Alternate domains are the union of reviewer-identified formative or independently originating lineages; later application settings alone are excluded. The evidence describes one principal historical lineage. Its defining controls and vocabulary remain bounded to a particular professional or technical practice. The encyclopedia entry generalizes the established mechanism without creating a new composite lineage.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
Its nearest twin is the Transactional Outbox/Inbox Pattern, because both fight duplication. The line between them: this store is a request-level lookup that short-circuits a duplicate call by key, whereas the outbox pattern is a messaging pattern whose core job is making a state change and its outbound event atomic — the outbox merely uses a dedup store like this one on its inbox side. Reach for this when the problem is "the same call arrives twice"; reach for the outbox when the problem is "did my commit and my message stay in sync."
[n1] An operation is idempotent when applying it many times has the same effect as applying it once. Distributed messaging typically guarantees only at-least-once delivery, so idempotency (here supplied externally by a dedup store keyed on the caller's identifier) is the standard way to recover effectively-once semantics — the approach popularized by payment APIs such as Stripe's idempotency keys. ↩