Deduplicating Union Pass¶
Batch operation — instantiates Inclusive Membership Union Design
Materializes the union set by emitting each canonical member exactly once, collapsing repeated appearances and folding their provenance into a single record.
Once every record carries a canonical key, the union still has to be built — and built without counting the same member twice. Deduplicating Union Pass is the batch step that walks the keyed records from all sources and emits one row per distinct member, no matter how many sources contributed it. Its defining move is collapse with memory: where several records share a key, it merges them into a single member yet keeps a folded record of every source that contributed, so nothing about origin is lost when the copies disappear. It takes for granted that the keys are already assigned and the membership decision already made; its own concern is the collapse policy — which fields survive a merge, how conflicts are broken, and how the merged provenance is carried forward.
Example¶
A listener wants one combined music library from their Spotify export and their Apple Music export. Roughly a third of the tracks exist in both. A naive concatenation produces a library where a third of the songs appear twice — once with each service's metadata. The deduplicating union pass walks the keyed track list and emits each canonical track once. Where the two exports disagree — Spotify says the album is the Deluxe Edition, Apple Music the Standard — a declared collapse policy decides the winner (prefer the more complete metadata, then the earlier release). Crucially, the surviving single row keeps a folded provenance stamp: sources: [spotify, apple_music], so the listener can still see the track came from both, even though only one copy remains.
The outcome is a library the same size as the true set of distinct songs, not the inflated sum of two overlapping lists — and every collapsed track still knows where it came from.
How it works¶
- Group by canonical key. Records that share a key form a collapse group; a group of one passes through untouched.
- Apply the collapse policy. For each group, a declared rule selects surviving field values — last-write-wins, most-complete-wins, trusted-source-wins, or a per-field merge — so the emitted member is deterministic rather than accidental.
- Fold, don't discard, provenance. The merge unions the contributing sources (and their versions and timestamps) into one provenance record attached to the surviving member; the copies vanish but their origins do not.
- Emit once. Each canonical key yields exactly one member in the output, so the union's size equals the count of distinct members — overlap is absorbed, never re-counted.
Tuning parameters¶
- Collapse rule — how a survivor is chosen when merged records disagree. Trusted-source-wins is predictable but discards good data from lesser sources; per-field merge keeps the best of each but can stitch together a member that no single source ever asserted.
- Merge grain — whether collapse operates on whole records or field by field. Field-level merging recovers more information but multiplies the conflict decisions and the ways a Frankenstein record can form.
- Provenance depth — how much origin detail the folded stamp keeps: just the source names, or names plus versions, timestamps, and the membership criterion each source applied. Deeper stamps cost storage but make the union explainable.
- Conflict logging — whether discarded values are dropped or retained for audit. Logging every override is heavy but lets a surprised consumer see what the collapse threw away.
When it helps, and when it misleads¶
Its strength is that it makes the union idempotent and honest: run it twice, feed it the same source twice, and the member count does not move, because union is idempotent — one appearance is as good as ten.[n1] That is exactly the property a merged list needs and a raw concatenation lacks.
Its failure mode lives in the collapse policy. A careless rule produces a merged record no source ever stated — a survivor whose title comes from one source, price from another, and category from a third, agreeing with nobody. The classic misuse is treating collapse as harmless cleanup and dropping the losing values silently, so a downstream analyst can never reconstruct why a member looks the way it does. The related trap is letting provenance evaporate in the merge, which erases the union's ability to answer "why is this member here?" The guarding discipline is to make the collapse rule explicit and deterministic, keep the folded provenance rich, and log overridden values rather than deleting them — so a collapse can always be explained and, if wrong, undone.
How it implements the components¶
duplicate_collapse_policy— its core: the declared rule that turns each group of same-keyed records into exactly one emitted member, with a defined survivor for every conflicting field.provenance_retention_record— the merge folds the contributing sources, versions, and timestamps into one stamp on the surviving member, so collapse never costs origin information.
This pass assumes the canonical keys already exist; producing them (canonical_member_identity_key) is Canonical Identity Resolution Pass, its nearest twin — resolution decides sameness, this pass acts on that decision. It also does not evaluate the at-least-one rule per candidate (inclusive_membership_rule); that is Inclusive-OR Membership Test.
Related¶
- Instantiates: Inclusive Membership Union Design — produces the materialized, dedup'd union set the archetype delivers.
- Consumes: Canonical Identity Resolution Pass supplies the canonical keys that define which records collapse together.
- Sibling mechanisms: Canonical Identity Resolution Pass · Inclusive-OR Membership Test · Overlap and Coverage Dashboard · Provenance Tagging Protocol · Type Compatibility Checklist · Union Delta Review · Union Specification Sheet
Editorial Notes¶
Form Classification¶
Form family: Intervention, Treatment & Transformation
Rationale: Deduplicating Union Pass operates as a direct treatment or transformation intended to change the target state or representation because it materializes the union set by emitting each canonical member exactly once, collapsing repeated appearances and folding their provenance into a single record.
Independent corroboration: The frozen evidence defines Deduplicating Union Pass as 'Materializes the union set by emitting each canonical member exactly once, collapsing repeated appearances and folding their provenance into a single record', so its operative form is Intervention, Treatment & Transformation.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Single lineage
Present-day reach: Specialized
Rationale: Data processing cohered keyed union passes that emit one canonical row per distinct member while folding source provenance and resolving field conflicts deterministically.
Related originating lineages:
- Mathematics — Set theory supplied idempotent union and the one-member-per-equivalence-class invariant.
Review resolution: Data processing cohered keyed union passes that emit one canonical row per distinct member while folding source provenance and resolving field conflicts deterministically.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Set union is idempotent: A ∪ A = A, and adding an already-present element changes nothing. A deduplicating pass makes a real dataset behave this way, so re-ingesting a source or running the pass twice cannot inflate the member count — the property a plain concatenation lacks. ↩