Leakage Scan¶
Validity audit — instantiates Task-Legible Feature Construction
Systematically interrogates a candidate feature set for information that would not be available in real use — future outcomes, post-decision fields, or forbidden proxies — before any of it ships.
A feature can be too good — good because it cheats. Leakage Scan is the dedicated audit that interrogates a whole candidate feature set for information it should not have: fields that encode the future, values populated only after the decision is made, columns that are near-copies of the label, and inputs that stand in for a forbidden or unstable attribute. Its defining property is that it hunts validity, not value — it asks "could this feature legitimately be known at decision time, and does it smuggle in something it shouldn't," never "does this feature improve the metric." The two questions pull in opposite directions: leaked features tend to look wonderful on the metric, which is exactly why a validity audit has to stand apart from any performance test. The scan is a gate the feature set passes before it is trusted, run against the decision-time reality of the consumer and, for proxy concerns, an expert reviewer.
Example¶
An insurer is building an underwriting model to price new auto policies. The feature set, assembled from several transformation mechanisms, scores suspiciously well on validation — a red flag, not a triumph. The leakage scan works through it systematically. First, decision-time availability: it maps each feature to when its value is actually populated, and catches claims_in_first_year — a field that only exists after the policy is written, an outright future leak. Second, label proximity: it flags manual_uw_decision_code, a near-restatement of the outcome the model is meant to predict. Third, proxy screening: it routes candidates to a compliance reviewer, who identifies that a granular zip9 feature is functioning as a proxy for a protected class in this jurisdiction — a legal and ethical problem no accuracy metric would ever surface.[n1]
None of these were caught by the mechanisms that built the features, because each looked locally fine. The scan removes the future-dated and label-proximate features outright, quarantines the proxy pending legal review, and clears the rest. The model's validation score drops — and is finally honest.
How it works¶
The scan is a structured interrogation, not a single test:
- Decision-time availability check. For each feature, establish exactly when its value exists relative to the prediction moment; anything populated at or after the decision is presumed leaked.
- Label-proximity check. Detect features suspiciously correlated with or derived from the target — near-duplicates, post-outcome codes, downstream artifacts.
- Proxy and fairness screen. Route features that could stand in for protected or forbidden attributes to a qualified reviewer, since proxy harm is a judgment, not a statistic.
- Quarantine, not just flag. Failing features are removed or held pending review before the set is trusted; the scan is a gate, not a dashboard.
The distinguishing act is auditing legitimacy against decision-time reality — deliberately blind to performance, because the leaked features are the high-performing ones.
Tuning parameters¶
- Availability strictness — how conservatively "known at decision time" is enforced. Strict removes borderline features (safe, may drop valid ones); lax risks live failure.
- Label-proximity threshold — how high a correlation with the target triggers scrutiny. Low thresholds catch subtle leaks but flood the reviewer with legitimate strong features.
- Proxy screen scope — which attributes count as forbidden and how far indirect proxies are chased. Broader screening reduces harm but can strip genuinely useful, legitimate signal.
- Gate hardness — hard block vs. warn-and-log. A hard gate stops leaks cold but can stall delivery on false alarms.
When it helps, and when it misleads¶
Its strength is catching the failures that reward you for having them — leaked and proxy features that inflate every offline number — which no performance-based check will ever flag, because performance is the symptom, not the alarm. It is the archetype's defense against target leakage, shortcut capture, and proxy discrimination.
Its failure mode is incompleteness masquerading as clearance: a scan that checks the obvious future-dated fields but misses an indirect leak (a feature correlated with collection timing, say) issues a clean bill of health that is worse than none, because it manufactures false confidence. The classic misuse is treating a passing scan as proof the features are good — it only proves they are legitimate; a valid feature can still be useless or unstable. And proxy screening done by rule alone, without expert judgment, both misses real proxies and flags innocent correlates. The guarding discipline is to keep the scan adversarial and evolving (new leak patterns get added as they are found), to keep the proxy screen human-reviewed, and to hand cleared features onward to a separate value test rather than assuming legitimacy implies worth.
How it implements the components¶
leakage_and_proxy_guardrail— it is the archetype's dedicated, cross-cutting realization of this guardrail: a systematic set-wide audit for leakage, label proximity, and proxy harm.downstream_consumer_profile— the availability check is defined by the consumer's real decision-time context: what is knowable when the prediction is made.domain_expert_review_gate— proxy and fairness judgments are routed to a qualified reviewer, because forbidden-substitute harm cannot be settled by a statistic.
This scan does NOT implement candidate_transformation_catalog — it builds no features, only audits them; construction belongs to siblings like Interaction Term Construction. Nor does it run the feature_validation_frame value test, which is Feature Ablation Comparison: the scan judges legitimacy, ablation judges worth.
Related¶
- Instantiates: Task-Legible Feature Construction — supplies the set-wide leakage and proxy gate.
- Sibling mechanisms: Feature Ablation Comparison · Categorical Encoding Scheme · Lag & Window Feature Extraction · Interaction Term Construction · Domain-Derived Feature Template · Normalization & Scaling Pipeline · Feature Importance & Stability Dashboard · Feature Store Versioning
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: The scan evaluates candidate features for unavailable future information, post-decision fields, and forbidden proxies and produces a validity finding.
Nearest alternative: Decision, Gate & Allocation — Leaking features may be blocked, but the defining output is the pre-release audit finding.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Data Science & Analytics
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Machine-learning validation developed systematic scans for future information, post-decision fields, and target-derived features.
Related originating lineages:
- Statistics & Experimental Design — Design-of-analysis practice supplied temporal availability and validity constraints.
- Ethics of Technology & AI Governance — Responsible-AI review materially shaped checks for forbidden proxies and deployment-inappropriate information.
Review resolution: Both independent reviews place the primary lineage in data_science. The queued differences (domain_reach_disagreement) concern secondary metadata rather than primary provenance. The final retains statistics_experimental_design, tech_ethics_ai_governance only where a reviewer supplied a formative-lineage rationale; downstream application by itself is not treated as origin. origin_mode=cross_disciplinary_synthesis records the relationship among origin traditions, while domain_reach=multi_domain records application breadth separately. encyclopedia_synthesis=false reflects whether either reviewer identified a corpus-specific synthesis, and confidence=high preserves the more cautious evidence assessment.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
The scan concentrates leakage judgment set-wide, but it does not have a monopoly on the concern: transformation mechanisms guard their own local leak surfaces — Categorical Encoding Scheme against target-encoding leakage, Lag & Window Feature Extraction against future-window leakage. The scan is the backstop that catches what slips past those local guards and, uniquely, screens for proxy harm across the whole set.
[n1] A proxy variable is a feature that stands in for another, often forbidden or protected, attribute — e.g., fine-grained geography acting as a proxy for a protected class. Proxy-based discrimination is well documented in insurance and lending (the historical practice of redlining is the canonical case), and detecting it requires domain and legal judgment, not correlation alone. ↩