Deduplication Workflow¶

Operational process — instantiates Equivalence Class Consolidation

A repeatable sweep over a defined population that groups records satisfying a duplicate criterion into clusters and collapses each cluster to one.

A Deduplication Workflow is the end-to-end operational pass that takes a whole population of records and returns it with the duplicates gathered and collapsed. Its distinguishing job is the sweep: it fixes exactly which records are in scope, runs them against each other, forms the actual duplicate clusters, and reduces each cluster to a single kept record. It is not the brain that decides whether two records match — it consumes that judgment — and it is not the vault that then builds a governed master record with full lineage. It is the repeatable process that turns "this dataset is riddled with duplicates" into "each real thing now appears once," over a bounded set, on a cadence.

Example¶

A mid-size charity's donor database has accreted the same supporter three times over a decade: "Robert Jones" at an old street address, "Bob Jones" at a newer one, and "R. Jones" from an online gift. The consequences are concrete — he gets three copies of every appeal, his lifetime giving is split across three records so he never crosses the threshold for a personal thank-you, and the year-end donor count is inflated. A deduplication workflow scopes the run (all individual constituents, excluding organizations), blocks them into comparable candidate sets by surname and postal code, clusters the records that a matching step flags as the same person, sends the genuinely ambiguous clusters to a fundraiser for a human call, and collapses each confirmed cluster into one supporter. Afterward Robert appears once, his true lifetime total is visible, and the mail house stops printing two wasted letters.

How it works¶

What sets this mechanism apart is that it operates at the level of a population, in batches, and produces clusters as its output:

It bounds the comparison scope first — which records, which segment, what time window — because comparing everything to everything does not scale; blocking partitions the population into candidate groups that are actually worth comparing.
It forms equivalence classes: the clusters of records judged to be the same real thing are the workflow's central product, each destined to be collapsed to one.
It routes by confidence, auto-collapsing high-certainty clusters and queuing borderline ones for human decision, then collapses each confirmed cluster to a surviving record.

Tuning parameters¶

Scope breadth — the whole database versus a segment or a rolling recent window. Wider scope catches more duplicates but multiplies comparisons and the chance of merging across contexts that should stay separate.
Blocking key — how records are partitioned before matching. A tight key is cheap but misses duplicates that disagree on the key; a loose key catches more but costs far more comparisons.
Auto-merge threshold — how sure the workflow must be before collapsing without human review. Loosening it clears the queue faster but converts uncertainty into silent false merges.
Merge destructiveness — hard-delete the losers versus a reversible soft merge that retains them. Reversibility is the safety net for the merges that turn out wrong.
Cadence — one-time cleanup, periodic batch, or continuous on-write. Continuous keeps the set clean; batch is cheaper and easier to audit.

When it helps, and when it misleads¶

Its strength is eliminating the duplicated work and distorted counts that fragmented records cause: one contact instead of three, an accurate total instead of a split one, a population you can actually reason about. Run periodically it keeps a growing dataset from silting up.

Its worst outcome is the false-positive merge — two genuinely different people fused into one record because the matching was too eager or the scope too broad, an error that is far harder to detect and undo after the fact than a missed duplicate.^[1] Over-broad scope compounds it by inviting merges across contexts where the equivalence never held. The classic misuse is optimizing for a deduplication rate — tuning the threshold loose to report "the file is 12% smaller" — which manufactures exactly those bad merges. The discipline is to keep merges reversible, hold ambiguous clusters for human decision rather than auto-collapsing them, and measure the workflow by match precision, not by how many records disappeared.

How it implements the components¶

comparison_scope — the workflow's first act is to define and bound the population it sweeps, and to block it into the candidate sets actually compared.
equivalence_class — the duplicate clusters it forms are the equivalence classes of the consolidation; producing and collapsing them is what the process does.

It does not define the sameness criterion or infer which records match — that inference is the Identity Resolution Model it consumes — and it does not build or govern the surviving master record with its aliases and history; that is Master Record Consolidation.

Instantiates: Equivalence Class Consolidation — the workflow is the operational sweep that finds and collapses duplicates across a population.
Consumes: Identity Resolution Model supplies the match decisions that tell the workflow which records belong in a cluster.
Sibling mechanisms: Identity Resolution Model · Master Record Consolidation · Alias Resolution Table · Canonicalization Pipeline · Crosswalk Table · Equivalence Test Suite · Policy Equivalence Rule · Synonym Merge Review · Taxonomy Merge Workshop · Unit Normalization Table

References¶

[1] In direct-marketing practice the batch process of matching and collapsing duplicate contacts across lists is called merge/purge; its long-known hazard is the over-merge that silently combines two distinct households, which is why reversible merges and precision-oriented tuning are standard safeguards. ↩