Duplicate Resolution Queue¶
Workflow — instantiates Exhaustive Population Mapping
Routes look-alike records to deterministic, probabilistic, and human adjudication so each real unit is counted exactly once.
A Duplicate Resolution Queue is the workflow that decides, one contested case at a time, whether two records that look like the same unit really are. Its defining act is adjudication under a decision rule: candidate pairs flow through a pipeline that resolves the easy ones automatically and escalates the genuinely ambiguous ones to human judgment, and every decision — merge, split, keep-apart — is committed as an auditable revision to the enumeration. Where a complete count fails by missing units, this mechanism guards the other failure: counting the same unit twice under different identifiers, which silently inflates totals and corrupts any per-unit action. It is not the store that holds the surviving records and not the timing rule for admitting brand-new units; it is the decision workflow that turns a pile of maybe-duplicates into clean, one-per-unit identity, and records why.
Example¶
An election office must ensure each eligible voter appears on the roll exactly once, because a duplicate registration can mean a double-counted ballot and a phantom name can mean a wrongful purge. When county rolls are pooled, thousands of near-matches surface: two "Robert J. Alvarez" records with the same birth date but different addresses; a "Katherine" and a "Kathy" Nguyen sharing an SSN fragment; a father and son with identical names one house apart. The queue routes them by confidence. Exact matches on a strong key (full name + date of birth + driver's-license number) auto-merge. Records agreeing on some fields and conflicting on others land in a clerical review queue for a human to judge.[n1]
The adjudicator's ruling is the point. The two Alvarez records are the same man who moved — merge, and log the address history so the merge can be reversed if wrong. The identically named father and son are distinct people — a deliberate keep-apart, recorded so a future automated pass doesn't re-merge them. Each decision is a versioned revision, not a silent overwrite, so a mistaken merge that disenfranchises a real voter can be traced and undone. The queue never invents a voter and never stores the final roll; it decides which records are one person and hands the clean identities onward.
How it works¶
- Score candidate pairs, don't pre-judge them. Incoming maybe-duplicates get a match confidence; the workflow's job is to route by that confidence, not to assume.
- Tier the decision. High-confidence pairs resolve deterministically; the ambiguous middle escalates to human clerical review; clear non-matches are dismissed and remembered.
- Commit merges and splits as revisions. Every consolidation or separation is a logged, reversible change to the enumeration, with the evidence attached.
- Feed decisions back. A keep-apart or a confirmed merge trains the thresholds so the same pair isn't re-litigated every cycle.
Tuning parameters¶
- Auto-merge threshold — how confident a match must be to merge without a human. Raise it and clerical volume balloons; lower it and wrong merges slip through.
- Escalation band width — how wide the "send to a human" middle is. A wide band is safer but expensive; a narrow one is cheap but riskier at the edges.
- Reversibility depth — whether merges keep full component history for un-merging. Deeper reversibility protects against error at a storage and complexity cost.
- Feedback aggressiveness — how strongly past adjudications auto-resolve future look-alikes. Strong feedback speeds throughput but can entrench an early mistake.
When it helps, and when it misleads¶
The queue matters most where a doubled unit does real harm — double billing, double benefits, double votes, double notifications — and where identity is genuinely ambiguous across sources. It converts an unbounded dedup problem into a governed, auditable stream of decisions, and its revision log is what keeps the enumeration corrigible rather than mysteriously changing.
Its two failure modes point in opposite directions and both are costly. Over-merging (a loose threshold) collapses distinct units — the identically named father and son become one, and one of them vanishes from action. Under-merging (a tight threshold) leaves duplicates that inflate the count. This is the false-match/false-non-match trade-off, and pretending a single threshold optimizes both is the classic error.[n1] A subtler trap is automation drift: an early wrong keep-apart or merge, fed back as training, silently propagates. The guarding discipline is to keep an escalation band wide enough that ambiguous cases meet a human, keep every merge reversible, and periodically re-check auto-resolved pairs rather than trusting the pipeline forever.
How it implements the components¶
A Duplicate Resolution Queue fills the decide-identity slice — the adjudication that makes one-per-unit true:
unit_identity_and_deduplication_key— decides, per candidate pair, whether two records are the same unit, producing the resolved identity that keys the enumeration.exception_and_revision_path— every merge, split, and keep-apart is committed as an auditable, reversible revision, with its evidence.
It decides identity but does not assemble or cross-reference the source registers that generate the candidates — the enumeration_frame_inventory and independent_source_crosswalk are Administrative Record Linkage's — nor does it persist the surviving records or guard their exposure, which are the unit_level_registry_or_map and privacy_and_exposure_guardrail held by Master Unit Index; and it consolidates existing records rather than admitting new units under a timing cutoff, which is the refresh_and_staleness_policy role of Late-Unit Inclusion Window.
Related¶
- Instantiates: Exhaustive Population Mapping — the queue is the adjudication that guards against duplicate inflation of the count.
- Consumes: Administrative Record Linkage supplies the candidate pairs the queue adjudicates.
- Sibling mechanisms: Census Protocol · Enumeration Area Map · Master Unit Index · Capture-Recapture Check · Door-to-Door or Field Sweep · Administrative Record Linkage · Coverage Gap Heatmap · Late-Unit Inclusion Window · Enumeration Quality Backcheck
Editorial Notes¶
Form Classification¶
Form family: Control, Automation & Runtime
Rationale: The live queue scores incoming candidate pairs and state-dependently routes them to deterministic merge, dismissal, or human review while feeding outcomes back into thresholds.
Nearest alternative: Decision, Gate & Allocation — Individual pairs receive dispositions, but the queue's operative form is a continuing executable routing system rather than a bounded human selection mechanism.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Data Science & Analytics
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Entity-resolution practice cohered match, nonmatch, and uncertain bands, routing ambiguous pairs to clerical adjudication rather than forcing one threshold.
Related originating lineages:
- Library & Information Science — Authority control supplied human review for variant names and duplicate catalog entities.
- Statistics & Experimental Design — Record-linkage methodology formalized the false-match versus false-non-match tradeoff that determines the clerical-review band.
Review resolution: Data science is primary because entity resolution supplies deterministic and probabilistic matching; statistics and library authority control shape confidence routing and governed human adjudication across many domains.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
The queue shares the exception_and_revision_path component with Late-Unit Inclusion Window, but their revisions are opposite in kind: the queue consolidates existing records (merging or splitting units already in view), while the inclusion window admits wholly new units after a cutoff. Same audited revision log, different direction of change.
[n1] In record-linkage and deduplication practice, pairs too uncertain to auto-classify are sent to clerical review — human adjudication of the ambiguous middle between confident match and confident non-match. The width of that middle band embodies the false-match / false-non-match trade-off: no single threshold minimizes both wrong merges and missed merges at once. ↩a ↩b