Duplicate Detection Audit¶
Audit — instantiates Birthday-Bound Collision Budgeting
Periodically sweeps assigned identifiers against a uniqueness registry to find and resolve duplicates that have already slipped in, rather than preventing them at assignment time.
The Duplicate Detection Audit is the after-the-fact counterpart to runtime prevention: it assumes some collisions have already happened — through a missing constraint, a merged partition, a bad import, or a filling space — and its job is to find them and clean them up. It maintains, or reads against, a uniqueness registry — an authoritative index of what has actually been assigned — and periodically sweeps the population for any value that appears more than once, then routes each duplicate down a resolution path: merge, re-key, escalate, or quarantine. Its defining trait is that it operates on the accumulated record, catching the duplicates that prevention missed and that no live check will ever notice on its own.
Example¶
A hospital merges the patient records of two departments that each issued their own medical record numbers (MRNs). Individually each system enforced uniqueness; together, the same MRN can now denote two different patients — precisely the partition-drift hazard the archetype warns about. A duplicate detection audit runs across the combined set: it builds a registry keyed on MRN, flags every number that resolves to more than one identity, and produces a worklist. Each conflict is routed for resolution — genuinely distinct patients are re-keyed under the merged scheme; true matches are reconciled into one record. The audit doesn't stop new collisions (that's for the assignment path); it finds the ones the merge silently created before a clinician pulls up the wrong chart.
How it works¶
- Assemble the registry. Build or read an authoritative index of assigned identifiers across the whole population being audited — the source of truth the sweep compares against.
- Sweep for multiplicity. Scan for any identifier that maps to more than one distinct entity, across partitions and time windows that share a comparison space.
- Route each finding. Send every confirmed duplicate down a resolution path — merge, re-key, quarantine, or escalate — with a record of the decision.
- Report the rate. Surface how many duplicates were found as a signal about whether prevention and sizing upstream are holding.
What distinguishes it from the retry protocol is timing and target: it works periodically on the accumulated record, not synchronously at the moment of assignment.
Tuning parameters¶
- Sweep cadence — continuous, nightly, or on-merge. More frequent sweeps shrink the window a duplicate survives undetected but cost more compute.
- Match definition — exact-identifier only, or fuzzy matching that also catches near-duplicates from encoding or human entry. Fuzzy catches more but risks false merges.
- Resolution default — auto-merge, auto-requeue, or human review per finding. Automation clears volume fast but a wrong auto-merge is hard to undo.
- Audit scope — one partition or the full cross-partition pool. Widening scope catches drift-induced collisions that a per-partition sweep would miss entirely.
When it helps, and when it misleads¶
Its strength is that it is the only mechanism that finds collisions already present — the residue of imperfect prevention, merges, and legacy data — and it gives a real measured collision rate to check the sizing assumptions against. Paired with a registry, it is also what lets a system claim uniqueness it can actually verify.
It misleads when its cadence lets duplicates live too long: a nightly audit still leaves a full day in which a duplicate MRN can send a clinician to the wrong chart, so detection latency is the residual risk. An over-eager fuzzy matcher can also merge records that should stay distinct, which is often worse than the duplicate it fixed. The classic misuse is leaning on the audit as the primary control while skipping sizing and prevention entirely — detecting collisions you could have prevented. The discipline is to size and prevent first, then run the audit as a backstop whose cadence matches the consequence tier.
How it implements the components¶
uniqueness_registry— maintains or reads the authoritative index of assigned identifiers that the sweep compares against, and that downstream checks can trust.collision_detection_and_resolution_path— the periodic sweep-and-resolve path that finds already-committed duplicates and routes each to merge, re-key, or escalation.
It works after the fact — the synchronous, at-assignment detect-and-regenerate loop and its retry_backoff_policy are Collision Retry Protocol's — and it does not watch live occupancy for early warning (growth_monitoring_trigger), which is Capacity Warning Dashboard's.
Related¶
- Instantiates: Birthday-Bound Collision Budgeting — the after-the-fact detection-and-cleanup backstop.
- Sibling mechanisms: Collision Retry Protocol · Capacity Warning Dashboard · Domain-Separated Identifier Scheme · Identifier-Space Capacity Check · Birthday-Bound Calculation · Collision Probability Table · Namespace Entropy Review · Hash Collision Risk Assessment · Adversarial Birthday-Attack Review
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Duplicate Detection Audit operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it periodically sweeps assigned identifiers against a uniqueness registry to find and resolve duplicates that have already slipped in, rather than preventing them at assignment time.
Independent corroboration: The frozen evidence defines Duplicate Detection Audit as 'Periodically sweeps assigned identifiers against a uniqueness registry to find and resolve duplicates that have already slipped in, rather than preventing them at assignment time', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Data Science & Analytics
Origin pattern: Convergent development
Present-day reach: Multi-domain
Rationale: Record-linkage and data-quality practice is primary because it established population-level matching, duplicate detection, and governed disposition of records that may represent the same or different entities.
Related originating lineages:
- Computer Science & Software Engineering — Database and namespace engineering supply authoritative uniqueness registries, multiplicity sweeps, and re-key operations.
- Library & Information Science — Authority-control practice independently established duplicate-record review, selection of the record to retain, and controlled deletion or merge.
Review resolution: Fellegi and Sunter formalized computer-oriented record linkage, while Library of Congress procedures explicitly govern duplicate authority-record resolution. The periodic population audit is therefore best placed in data science, with genuine convergent database and library lineages.
Attribution caveat: Statistical record linkage, database integrity, and catalog authority control independently produced recognizable duplicate-audit forms.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
- Fellegi and Sunter: A Theory for Record Linkage
- Library of Congress: Duplicate Series Authority Records—Resolution Procedures
Notes¶
The uniqueness registry this audit maintains is also what a Collision Retry Protocol checks against at assignment time — the same source of truth serves prevention and detection. Where guaranteed uniqueness is required outright, that registry (not the birthday budget) becomes the primary control, and the audit becomes its reconciliation arm.