Duplicate Target Scan¶
Test or assessment — instantiates Collision-Free Mapping Design
A scan that detects target values assigned to multiple distinct sources.
A duplicate target scan is a batch pass over existing data that hunts for target values already shared by two or more genuinely distinct sources — collisions that a live constraint never caught because they predate it, arrived through a bulk import that bypassed it, or crossed a boundary no single index spans. Its defining move is not detecting repeated values (that is trivial) but deciding which repetitions are collisions: it applies an explicit source-identity boundary to separate a benign duplicate (the same entity appearing twice) from a real collapse (two different entities wearing one identifier). Where a unique index guards one write at a time going forward, this scan looks backward and outward across a whole population and reports the damage already done.
Example¶
Two universities merge after a campus consolidation, and their student information systems must become one. Each legacy system minted eight-digit student IDs from its own counter, in blissful ignorance of the other, so the combined table almost certainly contains numbers claimed by two different people. A naive "group by student_id having count > 1" would drown the registrar in false alarms, because many students attended both campuses and legitimately appear twice under one shared record.
The scan applies the source-identity boundary the registrar actually cares about — same person means same national ID plus birthdate, not same eight-digit number — and uses it to split the pile. It finds 47 IDs held by genuinely distinct students, and for each it attaches both source records: the two names, two programs, two enrollment dates that collapsed onto one key. That evidence, not the bare count, is what lets the registrar renumber the true collisions before the student ID becomes the primary key everything else joins on.
How it works¶
- Run as a batch, not a gate. It sweeps existing, imported, or migrated data rather than intercepting a single write — the complement to a live constraint, not a replacement for it.
- Check against active and recently-retired values. A value freed but still referenced downstream is a live collision hazard, so the scan consults the namespace's retired set, not only its active rows.
- Group by normalized target, then classify by source distinctness. Repetitions are grouped; the source-identity predicate decides which groups are collisions versus expected duplication.
- Emit preimages, not counts. For each true collision it outputs the colliding source records, so the result is directly repairable rather than merely alarming.
Tuning parameters¶
- Scan scope — current import batch versus full history including retired values. Wider scope catches more but costs more and revisits settled data.
- Distinctness predicate strictness — which fields define "same source." This dial is the false-positive/false-negative split: loosen it and true collisions hide as duplicates; tighten it and duplicates masquerade as collisions.
- Normalization before grouping — how aggressively values are canonicalized (case, whitespace, encoding) before comparison; too much fuses distinct targets, too little misses variant spellings.
- Fuzzy-match blocking / threshold — for probabilistic source matching, how close two sources must be to count as the same entity.
- Cadence — one-shot at migration versus scheduled on every bulk load. Determines whether new imports silently reintroduce collisions.
When it helps, and when it misleads¶
Its strength is reaching the collisions that live guards structurally cannot: the ones already sitting in the data before the guard existed, or delivered by an ETL job that wrote straight past it. It turns "we think the merge is clean" into an enumerated, evidence-backed list.
Its honest failure mode is that it is a detector, not a fixer, and its verdicts inherit the precision/recall tradeoff of record linkage.[n1] Loosen the source-identity predicate and the scan cries collision on the same person recorded twice; tighten it and it waves through two people who differ only in a field it ignored. The classic misuse is running it once at migration and declaring victory, so the next quarterly import quietly reintroduces exactly what the scan cleaned up. The guarding discipline is to fix the distinctness predicate with the domain owners who know what "same" means, and to schedule the scan on every bulk load rather than treating it as a one-time cleanup.
How it implements the components¶
collision_detection_guard— it is the batch detection pass, sweeping existing and imported data for shared targets the live guard missed.source_identity_boundary— it applies the definition of "distinct source" to classify each repeated value as a collision or a benign duplicate.preimage_evidence_record— its output is the set of colliding source records per target, so the finding is repairable, not just countable.
It does not prevent collisions at write time (Unique Index Constraint, Booking Lock), govern value lifecycle (Namespace Reservation Table), or adjudicate and authorize merges (Collision Quarantine Queue); it hands its findings to those.
Related¶
- Instantiates: Collision-Free Mapping Design — it is the archetype's after-the-fact detection surface for collisions already in the data.
- Consumes: Namespace Reservation Table supplies which values are active versus retired, so the scan knows what to check against.
- Sibling mechanisms: Unique Index Constraint · Collision Quarantine Queue · Preimage Audit Log · Hash Collision Check · Deterministic ID Allocator · Booking Lock
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Duplicate Target Scan operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it a scan that detects target values assigned to multiple distinct sources.
Independent corroboration: The frozen evidence defines Duplicate Target Scan as 'A scan that detects target values assigned to multiple distinct sources', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Data Science & Analytics
Origin pattern: Convergent development
Present-day reach: Multi-domain
Rationale: Record linkage cohered scanning for distinct sources mapped to the same target value and adjudicating probable collisions under precision-recall tradeoffs.
Related originating lineages:
- Computer Science & Software Engineering — Database integrity supplied uniqueness constraints and collision detection over keys and assignments.
- Library & Information Science — Authority control independently established duplicate-target detection where distinct entities have been conflated under one authorized record.
Review resolution: Data science is primary because record-linkage logic distinguishes repeated targets from genuinely distinct sources; database and library authority practices converged independently on population sweeps and governed duplicate resolution.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] In record linkage / entity resolution, the tradeoff between precision (few false matches) and recall (few missed matches). Loosening a match rule to catch more true collisions also fuses more distinct sources; tightening it does the reverse. There is no setting that eliminates both errors. ↩