Source Collection Ingestion Workflow¶
Intake workflow — instantiates Inclusive Membership Union Design
Registers each contributing collection as a first-class source and defines how an absent, empty, or failed source is handled — so the union knows exactly what it is unioning.
Every union begins with a mundane but decisive question: which collections are being combined, and what happens when one of them does not show up? Source Collection Ingestion Workflow is the front door. It registers each contributing collection as a named, first-class source — with an identifier, an owner, a fetch method, and an update cadence — so the union is defined over an explicit, auditable roster rather than whatever files happened to be in a folder. Its defining move is treating source presence itself as data: an empty source and a missing source are not the same thing, and the workflow declares, up front, how each is handled. It does not judge whether a source's contents are type-compatible, and it does not decide membership; it establishes the roster and the rules for when the roster is incomplete.
Example¶
A county emergency-management office assembles a single evacuation contact roster by unioning three sources on each activation: the county's own resident registry, a Red Cross shelter-intake list, and a hospital discharge list. During a wildfire the Red Cross feed times out and returns nothing. The critical question is: does "nothing" mean no one needs a shelter (a real empty source) or the feed is down (a failed source)? The ingestion workflow answers by policy: a registered source that fails to fetch is marked unavailable and the last known snapshot is carried forward with a staleness flag, while a source that fetches successfully but returns zero rows is recorded as a genuine empty. The roster itself names all three sources, their owners, and their expected cadence, so the union built during the activation is defined over a known set — not silently over two sources because the third quietly vanished.
The result is a union that can always state exactly which sources it drew on and which were degraded — the difference, in an emergency, between "we contacted everyone" and "we missed a shelter's worth of people and never knew."
How it works¶
- Register each source. Assign every collection a stable id, owner, fetch method, and cadence in a source registry, so the union's inputs are named and governed, not ad hoc.
- Distinguish the three empties. On each fetch, classify the result as populated, genuinely empty (fetched, zero members), or unavailable (fetch failed) — and never collapse the last two into "0 members."
- Apply the absence policy. For an unavailable source, follow the declared rule: carry forward the last snapshot with a staleness flag, hold the whole union, or proceed and record the gap.
- Hand off a known roster. Emit the set of admitted sources and their status to the union build, so every downstream step knows exactly what was and was not included.
Tuning parameters¶
- Absence policy — what a failed source does to the union: carry-forward stale data, exclude and flag, or block the build entirely. Carry-forward keeps the union whole but can hide a real change; blocking is safe but brittle when any source can fail.
- Staleness tolerance — how old a carried-forward snapshot may be before it is treated as unavailable rather than current. Loose tolerance keeps unions running; tight tolerance catches silent rot but stalls more often.
- Registration strictness — whether only pre-registered sources may contribute, or new sources can be admitted on the fly. Strict registration keeps the roster governed; permissive intake is flexible but invites unvetted sources.
- Cadence alignment — whether sources are ingested on a shared schedule or each on its own. A shared snapshot makes the union coherent in time; independent cadences are convenient but mix members captured days apart.
When it helps, and when it misleads¶
Its strength is that it makes the union's inputs explicit and governed: you can always name the sources, see which were degraded, and reproduce the exact roster a given union was built from. Its treatment of the three empties is really an application of null semantics — the discipline, familiar from SQL's three-valued logic, of distinguishing "absent/unknown" from "known to be empty," because conflating them silently corrupts every count downstream.[n1]
Its failure mode is the silent shrink: a source fails, its absence is read as "it contributed nothing," and the union quietly loses a chunk of membership that everyone assumes is still there. The classic misuse is running the union on whatever sources happen to be reachable, with no registry, so a missing source leaves no trace at all. A subtler trap is over-eager carry-forward, which papers over a source that has genuinely gone dark for weeks. The guarding discipline is to register every source, always distinguish empty from unavailable, and surface degraded sources loudly — never letting a missing input masquerade as a real absence of members.
How it implements the components¶
source_collection_registry— its core: the governed roster of named sources, with ids, owners, fetch methods, and cadences, that defines what the union draws on.null_and_empty_source_policy— the declared handling of absent, empty, and failed sources, so a source outage is never silently read as zero members.
This workflow governs the roster and source presence; it does not audit whether a source's element types are compatible with the union (type_compatibility_gate) — that vetting is Type Compatibility Checklist. Declaring the admissible universe and downstream-use scope (union_universe_scope, downstream_union_scope_label) is Union Specification Sheet.
Related¶
- Instantiates: Inclusive Membership Union Design — establishes the registered sources and absence policy the whole union is built on.
- Consumes: Union Specification Sheet declares which sources and universe this workflow is meant to ingest against.
- Sibling mechanisms: Canonical Identity Resolution Pass · Deduplicating Union Pass · Inclusive-OR Membership Test · Overlap and Coverage Dashboard · Provenance Tagging Protocol · Type Compatibility Checklist · Union Delta Review · Union Specification Sheet
Editorial Notes¶
Form Classification¶
Form family: Protocol, Workflow & Routine
Rationale: Source Collection Ingestion Workflow operates as a repeatable ordered procedure or handoff sequence that coordinates action because it registers each contributing collection as a first-class source and defines how an absent, empty, or failed source is handled — so the union knows exactly what it is unioning.
Independent corroboration: The frozen evidence defines Source Collection Ingestion Workflow as 'Registers each contributing collection as a first-class source and defines how an absent, empty, or failed source is handled — so the union knows exactly what it is unioning', so its operative form is Protocol, Workflow & Routine.
Nearest alternative: Record, Log & Register — Source Collection Ingestion Workflow includes features of a persistent ledger, log, register, or case record that preserves history and traceability, but its defining operation is a repeatable ordered procedure or handoff sequence that coordinates action.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Library & Information Science
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Registering each collection as a source and defining absent or failed-source semantics is collection and metadata ingestion governance.
Related originating lineages:
- Computer Science & Software Engineering — Data pipelines operationalize union, null, empty, and failure behavior.
- Data Science & Analytics — Source identity and missingness determine valid downstream analysis.
- Organizational & Management Science — Ownership and exception procedures keep source onboarding accountable.
Review resolution: The blind reviewers agree that library_information_science is the primary origin and differ only on alternate origin disagreement, origin mode disagreement. I preserve every independently explained alternate from both records rather than imposing a numeric cap. I retain cross_disciplinary_synthesis because the combined evidence shows material contributions from several lineages. The broader reach of multi_domain records portability separately from historical provenance; encyclopedia_synthesis=true preserves the affirmative synthesis judgment where either reviewer identified one.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] SQL's three-valued logic distinguishes a value that is present, one that is known to be empty, and one that is NULL (absent/unknown) — and treats them differently in aggregates precisely so that "missing" is never silently counted as "zero." The ingestion workflow applies the same distinction to whole sources: unavailable is not empty. ↩