Co-Occurrence Weighting Pipeline¶
A batch statistical tool — instantiates Co-Activation Coupling Design
Counts how often units appear together inside a defined window and re-weights the raw tallies so that frequency artifacts don't masquerade as meaningful association.
Before any link should be strengthened, you need to know how surprising a co-occurrence is — because two units that appear together constantly may share nothing but popularity. The Co-Occurrence Weighting Pipeline is the batch tool that produces that evidence: it enumerates the set of units to be coupled, slides a window over the data to tally which units land together, and then re-weights the raw counts so that association strength reflects excess co-occurrence over chance rather than sheer frequency. Its defining move is the re-weighting step — turning a naïve count into a statistic that a hugely popular unit cannot dominate simply by being everywhere.
Example¶
A grocery chain wants to learn which products genuinely go together for a recommendation shelf. The pipeline defines the unit set (SKUs), sets a window (items in the same basket), and tallies co-occurrence across a season of transactions. Raw counts are useless on their own: bananas co-occur with everything, so by count they look "associated" with motor oil. The re-weighting step fixes this — dividing observed co-occurrence by what independence would predict (a lift/PMI-style adjustment[n1]). After re-weighting, bananas–motor-oil collapses toward 1 (no real association) while tortillas–salsa rises well above 1 (≈4× expected). The output is a table of weighted, window-scoped co-occurrence scores — the honest evidence a downstream update rule should step on, not the frequency mirage.
How it works¶
Three stages, in order. Define the coupled unit set and window — which entities can link, and what "together" means (same basket, a ±5-token span, the same session). The window choice silently decides half the result. Accumulate the correlation trace — a sparse tally of joint occurrences, plus each unit's marginal frequency, over the whole corpus rather than any single episode. Re-weight — normalize joint counts against the marginals so the score measures departure from chance, optionally shrinking low-count pairs whose scores are statistically noisy. The pipeline stops there: it emits evidence, not weights-in-a-live-matrix, and makes no per-event update.
Tuning parameters¶
- Window size / shape — wide windows find loose thematic links but blur precise ones; narrow windows capture tight pairings but miss slower relationships.
- Weighting scheme — raw count vs. lift vs. PMI vs. positive-PMI. Stronger normalization suppresses frequency artifacts but can over-amplify rare pairs.
- Low-count shrinkage — how aggressively to discount pairs seen only a handful of times, trading recall of genuine rare links against noise.
- Unit granularity — how finely the coupled unit set is carved (item vs. category), which sets both resolution and sparsity.
When it helps, and when it misleads¶
Its strength is that it separates popular from associated — the single most common way naïve co-activation designs go wrong. It gives every downstream mechanism a shared, auditable evidence base instead of anecdote. Its failure modes are all upstream-of-the-window: the wrong window size manufactures or hides associations wholesale, and no amount of clever weighting recovers a signal the window threw away. Sparse pairs stay unreliable no matter the scheme, and — the classic misuse — the pipeline is easily tuned after the fact until a desired pairing clears the threshold, which is fitting the evidence to the conclusion. The discipline is to fix window and weighting before looking at which pairs win, and to carry low-count uncertainty forward rather than reading a shaky score as a strong one.
How it implements the components¶
coupled_unit_set— it enumerates and scopes the units eligible to be linked; nothing downstream can couple a unit the pipeline didn't admit.correlation_trace— its central artifact: the accumulated, window-scoped record of repeated co-occurrence, timing, and reliability.co_activation_window— it operationalizes "jointly active" as a concrete window and applies it uniformly across the corpus.
It produces evidence, not weights: the per-event local_update_rule and the live link_strength_state belong to the Association Matrix Update Rule; the schedule of when pairs are presented (repetition_and_spacing_plan) belongs to the Temporal Contiguity Training Schedule; and detecting that a high-scoring pair is a shortcut (spurious_coupling_monitor) belongs to the Spurious Association Probe Set.
Related¶
- Instantiates: Co-Activation Coupling Design — this pipeline supplies the correlation evidence the whole design is supposed to be governed by.
- Sibling mechanisms: Association Matrix Update Rule · Decorrelation Separation Protocol · Competitive Inhibition Review · Context-Gated Pairing Exercise · Paired Activation Rehearsal Protocol · Pruning & Decay Maintenance Cycle · Replay Consolidation Window · Spurious Association Probe Set · Temporal Contiguity Training Schedule
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Counts how often units appear together inside a defined window and re-weights the raw tallies so that frequency artifacts don't masquerade as meaningful association, making its operative form a computation, comparison, model, or analytic representation used to infer, estimate, or choose.
Independent corroboration: The frozen evidence defines Co-Occurrence Weighting Pipeline as 'Counts how often units appear together inside a defined window and re-weights the raw tallies so that frequency artifacts don't masquerade as meaningful association', so its operative form is Analysis, Modeling & Optimization.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Data Science & Analytics
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Data mining established corpus-scale and transaction-scale pipelines that count joint occurrences and normalize them into association measures such as lift rather than treating popularity as evidence of linkage.
Related originating lineages:
- Linguistics & Semiotics — Corpus linguistics established windowed word co-occurrence weighted by mutual information to reveal association beyond raw frequency.
- Statistics & Experimental Design — Probability and association measurement contribute independence baselines, marginal normalization, shrinkage, and uncertainty for sparse pairs.
Review resolution: Church and Hanks directly established mutual-information weighting of lexical co-occurrence, and Agrawal, Imieliński, and Swami established large-scale association mining over baskets. Because the source mechanism explicitly covers both token windows and baskets and emits a general weighted association table, data science is primary while linguistics and statistical association remain formative lineages.
Attribution caveat: PMI-weighted word windows are a seminal computational-linguistics lineage, while market-basket association mining independently supplies the general transaction pipeline. The mechanism is deliberately domain-neutral, so data science is the broader primary.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
- ACL: Word Association Norms, Mutual Information, and Lexicography
- ACM SIGMOD: Mining Association Rules between Sets of Items in Large Databases
Notes¶
[n1] Pointwise mutual information (and its retail cousin lift) measure how much more often two units co-occur than independence would predict — log( P(i,j) / (P(i)·P(j)) ). This is the standard corrective for the "everything co-occurs with popular things" artifact, which is why it anchors the re-weighting step above. ↩