Skip to content

Source Collection Ingestion Workflow

Intake workflow — instantiates Inclusive Membership Union Design

Registers each contributing collection as a first-class source and defines how an absent, empty, or failed source is handled — so the union knows exactly what it is unioning.

Version
v2 · 2026-09-14 · History
Mechanism #
8528
Type
Intake Workflow
Form family
Protocol, Workflow & Routine
Solution family
Classification & Taxonomy
Problem family
Identity, Provenance & Integrity Failure
Problem subfamily
Collision, Membership & Feature Binding
Domain group
Professional & Organizational Practice
Origin domain
Library & Information Science
Also from
Computer Science & Software Engineering, Data Science & Analytics, Organizational & Management Science
Instantiates
Inclusive Membership Union Design

Every union begins with a mundane but decisive question: which collections are being combined, and what happens when one of them does not show up? Source Collection Ingestion Workflow is the front door. It registers each contributing collection as a named, first-class source — with an identifier, an owner, a fetch method, and an update cadence — so the union is defined over an explicit, auditable roster rather than whatever files happened to be in a folder. Its defining move is treating source presence itself as data: an empty source and a missing source are not the same thing, and the workflow declares, up front, how each is handled. It does not judge whether a source's contents are type-compatible, and it does not decide membership; it establishes the roster and the rules for when the roster is incomplete.

Example

A county emergency-management office assembles a single evacuation contact roster by unioning three sources on each activation: the county's own resident registry, a Red Cross shelter-intake list, and a hospital discharge list. During a wildfire the Red Cross feed times out and returns nothing. The critical question is: does "nothing" mean no one needs a shelter (a real empty source) or the feed is down (a failed source)? The ingestion workflow answers by policy: a registered source that fails to fetch is marked unavailable and the last known snapshot is carried forward with a staleness flag, while a source that fetches successfully but returns zero rows is recorded as a genuine empty. The roster itself names all three sources, their owners, and their expected cadence, so the union built during the activation is defined over a known set — not silently over two sources because the third quietly vanished.

The result is a union that can always state exactly which sources it drew on and which were degraded — the difference, in an emergency, between "we contacted everyone" and "we missed a shelter's worth of people and never knew."

How it works

  • Register each source. Assign every collection a stable id, owner, fetch method, and cadence in a source registry, so the union's inputs are named and governed, not ad hoc.
  • Distinguish the three empties. On each fetch, classify the result as populated, genuinely empty (fetched, zero members), or unavailable (fetch failed) — and never collapse the last two into "0 members."
  • Apply the absence policy. For an unavailable source, follow the declared rule: carry forward the last snapshot with a staleness flag, hold the whole union, or proceed and record the gap.
  • Hand off a known roster. Emit the set of admitted sources and their status to the union build, so every downstream step knows exactly what was and was not included.

Tuning parameters

  • Absence policy — what a failed source does to the union: carry-forward stale data, exclude and flag, or block the build entirely. Carry-forward keeps the union whole but can hide a real change; blocking is safe but brittle when any source can fail.
  • Staleness tolerance — how old a carried-forward snapshot may be before it is treated as unavailable rather than current. Loose tolerance keeps unions running; tight tolerance catches silent rot but stalls more often.
  • Registration strictness — whether only pre-registered sources may contribute, or new sources can be admitted on the fly. Strict registration keeps the roster governed; permissive intake is flexible but invites unvetted sources.
  • Cadence alignment — whether sources are ingested on a shared schedule or each on its own. A shared snapshot makes the union coherent in time; independent cadences are convenient but mix members captured days apart.

When it helps, and when it misleads

Its strength is that it makes the union's inputs explicit and governed: you can always name the sources, see which were degraded, and reproduce the exact roster a given union was built from. Its treatment of the three empties is really an application of null semantics — the discipline, familiar from SQL's three-valued logic, of distinguishing "absent/unknown" from "known to be empty," because conflating them silently corrupts every count downstream.[1]

Its failure mode is the silent shrink: a source fails, its absence is read as "it contributed nothing," and the union quietly loses a chunk of membership that everyone assumes is still there. The classic misuse is running the union on whatever sources happen to be reachable, with no registry, so a missing source leaves no trace at all. A subtler trap is over-eager carry-forward, which papers over a source that has genuinely gone dark for weeks. The guarding discipline is to register every source, always distinguish empty from unavailable, and surface degraded sources loudly — never letting a missing input masquerade as a real absence of members.

How it implements the components

  • source_collection_registry — its core: the governed roster of named sources, with ids, owners, fetch methods, and cadences, that defines what the union draws on.
  • null_and_empty_source_policy — the declared handling of absent, empty, and failed sources, so a source outage is never silently read as zero members.

This workflow governs the roster and source presence; it does not audit whether a source's element types are compatible with the union (type_compatibility_gate) — that vetting is Type Compatibility Checklist. Declaring the admissible universe and downstream-use scope (union_universe_scope, downstream_union_scope_label) is Union Specification Sheet.

Editorial Notes

Form Classification

Form family: Protocol, Workflow & Routine

Rationale: Source Collection Ingestion Workflow operates as a repeatable ordered procedure or handoff sequence that coordinates action because it registers each contributing collection as a first-class source and defines how an absent, empty, or failed source is handled — so the union knows exactly what it is unioning.

Independent corroboration: The frozen evidence defines Source Collection Ingestion Workflow as 'Registers each contributing collection as a first-class source and defines how an absent, empty, or failed source is handled — so the union knows exactly what it is unioning', so its operative form is Protocol, Workflow & Routine.

Nearest alternative: Record, Log & Register — Source Collection Ingestion Workflow includes features of a persistent ledger, log, register, or case record that preserves history and traceability, but its defining operation is a repeatable ordered procedure or handoff sequence that coordinates action.

Review outcome: Independent reviewer agreement; medium confidence.

Origin Attribution

Primary origin: Library & Information Science

Origin pattern: Cross-disciplinary synthesis

Present-day reach: Multi-domain

Rationale: Registering each collection as a source and defining absent or failed-source semantics is collection and metadata ingestion governance.

Related originating lineages:

Review resolution: The blind reviewers agree that library_information_science is the primary origin and differ only on alternate origin disagreement, origin mode disagreement. I preserve every independently explained alternate from both records rather than imposing a numeric cap. I retain cross_disciplinary_synthesis because the combined evidence shows material contributions from several lineages. The broader reach of multi_domain records portability separately from historical provenance; encyclopedia_synthesis=true preserves the affirmative synthesis judgment where either reviewer identified one.

Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.

Review outcome: Reconciled after independent review; high confidence.

References

[1] Codd, E. F. "Extending the Database Relational Model to Capture More Meaning". ACM Transactions on Database Systems 4(4), 397–434 (1979). Introduces three-valued database logic in which comparisons involving null values can evaluate to unknown rather than true or false. registry ↩