Data Leakage Audit¶

Provenance audit — instantiates Shortcut-Reliance Mitigation

Traces the provenance of every feature and split to catch information that leaks from the future, the label, or duplicated rows into training or validation — and records where each leak entered.

The cheapest possible shortcut is being handed the answer. Data Leakage Audit follows the data pipeline backwards to find it — the one mechanism that treats the provenance of each feature and split as the object of scrutiny rather than the model's behavior. For every feature it asks whether the value could legitimately be known at prediction time and whether it derives from the label or the future; for every split it asks whether test information contaminated training through duplicates, entity overlap, or temporal bleed. Its defining move is following data lineage back to source tables and timestamps, because leakage is a property of how the data was assembled, not of what the model learned. It answers a question no behavioral test can — "is this signal one the model should even have had?" — and records, for each leak, exactly where it got in.

Example¶

A team's customer-churn model scores an AUC around 0.99, which is too good to be trusted. A leakage audit traces each feature to its source. It finds days_since_last_login is computed as of the churn date rather than the prediction date, so it silently encodes the outcome; a support_ticket_resolved flag is only ever set after an account has already been marked for closure, another peek at the label. Tracing the split, it finds a batch of customers duplicated across train and test from a re-export, so the model was partly graded on rows it had memorized. Each finding is written to a lineage record — source table, computation timestamp, and the join where it entered. With the leaked features removed and the split de-duplicated, performance falls to a believable AUC near 0.78. That lower number is the honest one; the audit's value was exposing that the original score measured recall of the answer key, not prediction.

How it works¶

Interrogate every feature against the prediction clock. For each input, ask whether its value would be knowable at the moment a real prediction is made — anything computed as-of the outcome or the future is leakage.
Trace derivation to source. Follow each feature back through joins, aggregations, and resampling to the tables and timestamps it came from, rather than trusting its column name.
Screen the split for contamination. Check for duplicated rows, the same entity in train and test, and temporal bleed across the cut — the ways a test set stops being held out.
Record a lineage entry per finding. Each leak is logged with its source and entry point, so the fix is verifiable and the same channel can be re-checked later.

Tuning parameters¶

Provenance depth — spot-checking the top features versus tracing full lineage for all of them. Full tracing catches subtle leaks but is labor-intensive.
Temporal strictness — how hard the "as-of prediction time" cutoff is enforced. Strict cutoffs catch look-ahead leakage but can reject features that are merely awkward to time correctly.
Contamination scope — whether the split check covers row duplicates only, or also entity overlap and group/temporal leakage. Wider scope catches more but requires knowing the entity and time structure.
Trigger — a routine pre-training audit versus one fired only when metrics look implausibly high. Routine auditing prevents surprises; reactive auditing spends effort only when warranted but arrives late.
Automation — manual tracing versus pipeline-instrumented lineage. Instrumentation scales and re-runs cheaply; manual tracing catches the one-off join a tool won't model.

When it helps, and when it misleads¶

Its strength is catching the most common cause of "too good to be true" — a leak that no behavioral robustness test will flag, because the model genuinely is accurate on data that carries the same leak. It is the fastest explanation for a suspiciously high score and the cheapest to act on, since the fix is usually to drop a feature or repair a split.

Its limit is that it only catches the leaks it thinks to trace: a narrowly scoped audit misses subtle entity or temporal leakage, and a clean bill of health can breed overconfidence. The classic misuse is running it only after a great score, narrowly, to rationalize the number rather than to genuinely trace provenance — a checkbox audit scoped to pass. The discipline that guards against this is to audit before trusting any score, to trace lineage end-to-end rather than sampling, and to treat implausibly high performance as a leakage prior rather than a triumph. The failure it targets is classically called target leakage.^[1]

How it implements the components¶

Data Leakage Audit fills the provenance side of the archetype — the components about where the data came from, not what the model does with it:

artifact_and_leakage_screen — the systematic sweep of the pipeline for leaked, label-derived, future, or duplicated information is this screen.
shortcut_lineage_record — each finding is logged with its source table, timestamp, and entry point, tracing the leak's provenance so the fix is verifiable and re-checkable.

It does NOT enumerate behavioral cues by adversarial review (Artifact Red-Team Review); it does NOT judge whether a legitimately-available feature is causally meaningful (Causal Feature Review Panel); and it does NOT test the model's reliance on any cue (Counter-Correlated Holdout Set).

Instantiates: Shortcut-Reliance Mitigation — the audit rules out the degenerate case where the model was simply handed the answer.
Sibling mechanisms: Artifact Red-Team Review · Causal Feature Review Panel · Counter-Correlated Holdout Set · Challenge-Set Refresh Cycle · Deployment Canary and Drift Sentinel · Domain-Shift Stress Test · Feature Ablation or Occlusion Test · Group-Stratified Validation · Invariance Probe · Hard-Negative Data Augmentation · Shortcut-Risk Model Card Section

Notes¶

Leakage is a data problem, not a model problem — no amount of behavioral testing substitutes for tracing provenance, because a leaked model is accurate on every test that carries the same leak. Conversely, a clean audit does not rule out shortcuts learned on legitimately-available cues; that is the job of the decorrelation tests, which start where this one ends.

References¶

[1] Target leakage — when information derived from the label, or unavailable at prediction time, enters the features or the training/validation split, inflating measured performance in a way that vanishes in deployment. Because the model is truly accurate on the leaked data, the defect is only visible by auditing data provenance, not by testing model behavior. ↩