Deduplication Table or Ledger¶

Artifact — instantiates Idempotent Operation Design

A persisted record of seen operation identities, completion status, and results used to detect and resolve duplicates.

Version: v1 · 2026-08-24 · History
Mechanism #: 2517
Type: Artifact
Form family: Record, Log & Register
Solution family: Ordering, Sequencing & Dependencies
Problem family: Identity, Provenance & Integrity Failure
Problem subfamily: Execution-Time Referent & Repeat Integrity
Origin domain: Computer Science & Software Engineering
Instantiates: Idempotent Operation Design

When an operation cannot be reduced to a target state, the system has to remember what it has already done. Deduplication Table or Ledger is that memory: a persisted record, one row per distinct operation identity, holding whether the operation completed, what result it produced, and when. It is the passive artifact the rest of the archetype consults — a mechanism looks up an incoming identity, finds a row (or doesn't), and acts accordingly. Its defining trait is that it stores and recognizes but does not itself decide: it can tell you "this identity has been seen and here is its recorded outcome," yet it holds no opinion about what target the operation should reach, what to return to the caller, or which side effects to suppress. Those judgments belong to other mechanisms; the ledger is the shared substrate they all read from.

Example¶

A public-benefits agency processes housing-assistance applications submitted through a web portal, a call center, and paper mail. The same household frequently applies twice — a resident who never got a confirmation letter mails a duplicate a week after submitting online. Without a shared record, each channel opens its own case, and the household ends up with two case numbers, two review timelines, and conflicting notices. The agency introduces a deduplication ledger keyed on a stable operation identity (a hash of applicant identifiers plus program and benefit period). Every intake writes to it: if the identity is new, a row is created with status received; if it already exists, the incoming submission is matched to the existing case and logged as a duplicate attempt rather than a new case. The ledger also retains when and through which channel each attempt arrived, so a caseworker reviewing the file can see the full history — three submissions, one case — and explain to the applicant that their application was received the first time.

How it works¶

The artifact does four things and stops there:

Assign and store operation identity. Each operation gets a durable identifier — a key, fingerprint, or composite of business fields — written as the row's primary key so the same identity maps to one row.
Detect on lookup. An incoming attempt is checked against the table; a present identity is a duplicate, an absent one is new. This lookup is the duplicate detector.
Record completion. The row carries status (received, in-progress, completed, failed) and, where relevant, the produced result, so later readers know not just that it was seen but how it ended.
Preserve the history. Attempts, matches, and suppressions are appended, giving an audit trail that lets reviewers reconstruct what happened without re-running anything.

Tuning parameters¶

Identity key breadth — narrow (exact request key) versus broad (actor + target + period). A narrow key detects only literal repeats; a broad key catches semantic duplicates but risks colliding genuinely distinct operations.
Retention window — how long rows persist before archival. Long retention catches far-apart duplicates and satisfies audit needs but grows storage and privacy exposure; short retention lets late duplicates escape as "new."
Status granularity — a mere seen/not-seen bit versus a full lifecycle with results. Richer status supports reconciliation and replay but couples the ledger to more of each operation's detail.
Write timing — record identity before executing (reserve) versus after completing. Reserving first closes the concurrent-duplicate race; recording after is simpler but leaves a window where two attempts both look new.
Isolation of the check-and-insert — atomic (unique constraint / transaction) versus best-effort. Atomic prevents two racing inserts; best-effort is cheaper but reopens the duplicate.

When it helps, and when it misleads¶

Its strength is that it is the one durable place duplicates can be recognized across channels, restarts, and time — a shared record is what lets an idempotent receiver work even when the operation itself keeps no memory.^[n1] It is the substrate the whole archetype leans on when convergence-by-construction is not available.

It misleads when treated as if storing were the same as solving. A ledger that faithfully records identities still does nothing on its own about side effects — the row can say "completed" while a duplicate confirmation quietly dispatches a second shipment, because suppressing that effect is a different mechanism's job. Its two sharp failure modes are key collision (too broad a key suppresses a legitimate distinct operation as a duplicate, the more dangerous error because the loss is silent) and late-duplicate escape (a retention window shorter than the real retry horizon lets an old attempt reappear as new). The guarding discipline is to bind the identity key to enough context to separate true duplicates from new intent, size retention to observed retry and replay behavior, and remember that the ledger enables the resolution without being it.

How it implements the components¶

Deduplication Table or Ledger fills the memory-and-recognition subset of the archetype — the components that let repeats be seen:

operation_identity — it assigns and persists the identifier that marks what counts as the same intended operation, one row per identity.
duplicate_detection — the lookup against stored identities is the detector: present means repeat, absent means new.
completion_record — each row holds whether the operation completed and what result it produced, the evidence later mechanisms consult.
audit_trail — it preserves the history of attempts, matches, and suppressions for later review without re-execution.

It stores and recognizes but does not act: it does not decide the target condition (target_state — Upsert or Set Operation), choose what to return to a duplicate caller (result_replay_policy — Cached Result Replay, its nearest twin: the ledger holds the recorded result, while Cached Result Replay is the policy that hands that result back), or suppress outbound side effects (side_effect_guard — Outbox Deduplication).

Instantiates: Idempotent Operation Design — it is the durable record that lets duplicates be recognized when they cannot be dissolved.
Sibling mechanisms: Upsert or Set Operation · Cached Result Replay · Outbox Deduplication · Checklist Confirmation · Duplicate-Safe Payment Operation · Idempotent API · Safe Retry Protocol · Event Replay Deduplication

Editorial Notes¶

Form Classification¶

Form family: Record, Log & Register

Rationale: Deduplication Table or Ledger operates as a durable record, ledger, register, or trace whose value depends on preserving actual state or history because it a persisted record of seen operation identities, completion status, and results used to detect and resolve duplicates.

Independent corroboration: The frozen evidence defines Deduplication Table or Ledger as 'A persisted record of seen operation identities, completion status, and results used to detect and resolve duplicates', so its operative form is Record, Log & Register.

Review outcome: Independent reviewer agreement; high confidence.

Origin Attribution¶

Primary origin: Computer Science & Software Engineering

Origin pattern: Single lineage

Present-day reach: Specialized

Rationale: Enterprise integration and distributed systems cohered persistent deduplication stores recording operation identities, completion status, and results so receivers can recognize repeats and preserve idempotency.