Conjunctive Filter Pipeline¶
Processing pipeline — instantiates Shared Subset Intersection Mapping
Realizes the intersection as an ordered chain of filters, narrowing a base population one collection's membership rule at a time.
Conjunctive Filter Pipeline computes the same common subset a symmetric query would, but it does so procedurally: it starts from a base population and passes it through a sequence of filter stages, each stage keeping only the members that satisfy the next collection's membership rule. Where a set query asks a single all-of-these question, the pipeline builds the answer as a funnel — after every stage the surviving population is smaller, and the final stage's survivors are exactly the elements that passed every rule. Its defining virtue is that each stage is a named, inspectable step: you can watch the population fall from one filter to the next and see precisely where members are lost. This makes it the natural implementation when the collections are streams or rule-based predicates rather than pre-materialized sets.
Example¶
A retailer's growth team wants the audience for a high-value re-engagement email. The rule: a contact must be an active subscriber, AND have opened at least one email in 30 days, AND live in a shipping region where the promoted product is stocked, AND not be enrolled in a competing loyalty tier. Each condition lives in a different system.
The pipeline registers the four collections as ordered stages and streams the full 2.1M-contact base through them. Stage one — active subscription — drops the population to 1.4M. Stage two — recent open — to 380k. Stage three — in-region — to 210k. Stage four — not in the competing tier — to 194k. The output is the 194k audience, and because each stage is logged, the team can see at a glance that the recent-open rule is doing most of the narrowing. If a stakeholder later argues the audience is too small, the pipeline shows exactly which predicate to relax. The order of stages is arranged for efficiency — cheapest and most-selective filters first — but the surviving set is the same regardless of order, because AND is order-independent.
How it works¶
The pipeline treats each participating collection as a membership predicate and composes them with logical AND. A base population enters the first stage; each stage forwards only the members its predicate accepts; the last stage's output is the intersection. Two things distinguish this realization from a symmetric query. First, it is staged and observable: every intermediate population is a real, countable artifact, which turns the intersection into a legible funnel. Second, it is predicate-native: a collection need not be a materialized list — it can be a live rule ("opened in the last 30 days") evaluated on the fly, which is why the pipeline suits streaming and event data. The ordering of stages is a performance choice only; putting the most selective filter first shrinks the work downstream without changing the result.
Tuning parameters¶
- Stage order — which predicate runs first. Most-selective-first minimizes total work; the result is invariant, so this is purely an efficiency dial.
- Predicate strictness per stage — how tight each rule is set. Relaxing one stage widens the funnel at that point; the funnel makes the coverage cost of each rule directly visible.
- Base population — what enters stage one. Too narrow a base can exclude true members before any filter runs; too broad wastes compute.
- Early-exit vs. full-scan — stop as soon as a member fails one predicate, or evaluate all predicates for full per-stage diagnostics. Early exit is faster; full scan tells you how many rules each dropped member failed.
- Batch vs. streaming — run over a fixed snapshot or continuously over a live event stream.
When it helps, and when it misleads¶
Its strength is legibility: the falling counts at each stage make the intersection auditable and turn "why is the result so small?" into a locatable answer. It is the practical embodiment of predicate pushdown — evaluating the cheapest, most-selective condition as early as possible to shrink the working set.[n1]
Its failure mode is that a badly ordered or badly scoped base can mislead about coverage even though the final set is correct: if the base population omits legitimate candidates, no downstream filter can recover them, and the tidy funnel hides the loss that happened before stage one. A related misuse is reading the intermediate counts as if they were independent — treating "380k passed the open filter" as the size of an audience, when three more filters still apply. The guarding discipline is to fix and document the base population deliberately, and to read only the final stage as the intersection, using the intermediate counts for diagnosis rather than as results.
How it implements the components¶
collection_scope_registry— it names and orders the participating collections as explicit pipeline stages, each with its source and version.membership_predicate_set— each stage is a membership predicate, applied as a filter; the pipeline is where those rules actually run.intersection_operation_rule— composing the stages with AND realizes the all-of-these operation: only members passing every predicate survive.
It does not build the shared_universe_and_identity_basis its stages rely on, nor does it govern the common_member_result_set as a durable artifact — those belong to identity_key_normalization and to its nearest twin, n_way_intersection_query, which evaluates every membership condition simultaneously as one symmetric set operation rather than as a sequential funnel.
Related¶
- Instantiates: Shared Subset Intersection Mapping — this pipeline is the procedural, stage-by-stage way to compute the common subset.
- Consumes: membership_predicate_audit certifies the predicates each stage applies; identity_key_normalization makes the streamed members comparable.
- Sibling mechanisms: n_way_intersection_query · membership_predicate_audit · intersection_cardinality_dashboard · empty_result_review_gate
Editorial Notes¶
Form Classification¶
Form family: Control, Automation & Runtime
Rationale: Realizes the intersection as an ordered chain of filters, narrowing a base population one collection's membership rule at a time, making its operative form a live operational control that automatically routes, enforces, adapts, or responds during execution.
Independent corroboration: The frozen evidence defines Conjunctive Filter Pipeline as 'Realizes the intersection as an ordered chain of filters, narrowing a base population one collection's membership rule at a time', so its operative form is Control, Automation & Runtime.
Nearest alternative: Protocol, Workflow & Routine — Each predicate automatically admits or drops members in a live staged pipeline, making execution control more central than the stage sequence.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Database and stream-processing practice established ordered predicate pipelines that realize logical conjunction by successive filtering.
Review resolution: Both reviewers agree on computer_science as primary. The defining operation is an ordered pipeline of conjunctive input filters; set intersection is mathematical substrate rather than an independently originating workflow lineage, and current uses span many software-mediated domains.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Predicate pushdown is a query-optimization technique that moves filtering conditions as close to the data source as possible and evaluates the most selective ones first, so later stages process far fewer rows. The conjunctive pipeline's most-selective-first ordering is the same idea applied by hand. ↩