Feature Availability Audit¶
Audit procedure — instantiates Leakage-Resistant Validation Design
Walks every candidate input and asks whether its value would truly have been known at decision time, cataloguing the fields that would not.
Before a single split is drawn, some features are already poisoned: they carry information that only exists because the outcome happened. The Feature Availability Audit is the review that finds them. Feature by feature, it fixes the decision-time boundary — the exact instant the model must act — and asks of each candidate input, "would this value, with this content, have been populated and correct at that instant?" Its output is a map, not a fix: a timeline of when each feature becomes knowable and an inventory of the concrete ways contamination enters. It is the only mechanism here that catches leakage a split can never repair, because a feature that encodes the future leaks in every fold.
Example¶
A hospital team building an early-warning model for patient deterioration lists a promising predictor: whether a broad-spectrum antibiotic was ordered. In backtest it is enormously informative. The audit walks it against the decision-time boundary — the moment the alert must fire, hours before deterioration is confirmed — and asks when that field is actually populated. The answer: the antibiotic is ordered after a clinician already suspects sepsis, i.e., downstream of the very event being predicted. The feature is flagged as post-outcome and pulled. Two others are reclassified as available-but-lagged (a lab result that lands ~90 minutes late) and routed to the As-Of Join Rule with an embargo rather than dropped. The audit produces no model and no metric — it produces a defensible boundary, a per-feature availability timeline, and a short list of exactly how leakage would otherwise have entered.
How it works¶
It is a procedural review, not a statistical test, and its distinctive step is disposition-by-timing:
- Pin the decision-time boundary precisely — the event that starts the clock (order placed, case opened, alert due).
- For each candidate feature, trace it to its source system and establish when it is populated and whether its content is stable or gets restated.
- Classify each into a disposition: available, available-but-lagged (usable with an embargo), post-outcome (populated only after the event), or target-derived (a restatement of the label).
- Record the survivors as a timeline and the rejects as an inventory of leak pathways, so the same trap cannot quietly return with the next feature.
Tuning parameters¶
- Boundary strictness — how sharply the decision instant is drawn. A fuzzy boundary ("some time on the application day") lets lagged fields slip through as if instantaneous.
- Evidence standard — take the schema's documentation at face value, or trace each field to its real populate-time. Documentation is faster and routinely wrong.
- Scope / depth — audit every field, or only the high-importance and suspicious ones. Narrow scope is cheaper but misses a sleeper.
- Disposition granularity — a blunt keep/drop, or a keep-with-embargo class that hands lagged features to an as-of join instead of discarding signal.
- Cadence — a one-time design-phase pass, or a re-audit triggered whenever a new feature is added to the pipeline.
When it helps, and when it misleads¶
Its strength is reach: it catches target leakage and post-outcome fields upstream of every split, which is the one class of leak that resampling, grouping, and temporal holdouts are all powerless against.
Its weakness is that it trusts semantics and documentation, so it misses purely statistical leakage that isn't visible from a field's meaning — an innocuous-looking feature that happens to correlate with the label through a data artefact. It also inherits auditor optimism. The classic misuse is the rubber-stamp pass run to bless features that were already built and shipped. The guarding discipline is to define the enemy explicitly as target leakage[1] — information available in training but not at prediction time — and to pair the semantic audit with empirical probes rather than treating a clean review as proof.
How it implements the components¶
decision_time_information_boundary— it fixes where the boundary sits and tests every candidate feature against it; this is the audit's founding act.feature_availability_timeline— its surviving output is the per-feature timeline of when each value becomes knowable and usable.leakage_pathway_inventory— the rejected features become an enumerated inventory of concrete leak routes (post-outcome fields, restatements, target-derived columns).
It maps the boundary but does not enforce it in data assembly — that is the As-Of Join Rule — and it reasons from field semantics, not from data, so it does not detect statistical or duplicate-based leakage, which is Duplicate and Near-Duplicate Scan.
Related¶
- Instantiates: Leakage-Resistant Validation Design — the audit produces the boundary and pathway map the rest of the design enforces.
- Sibling mechanisms: As-Of Join Rule · Duplicate and Near-Duplicate Scan · Entity-Grouped Split · Benchmark Deduplication Scan · Fresh Holdout Retest · Label Proxy Screen · Leakage Ablation Test · Time-Based Holdout · Preprocessing Fit-on-Training-Only · Nested Cross-Validation · Holdout Access Log
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Feature Availability Audit operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it walks every candidate input and asks whether its value would truly have been known at decision time, cataloguing the fields that would not.
Independent corroboration: The frozen evidence defines Feature Availability Audit as 'Walks every candidate input and asks whether its value would truly have been known at decision time, cataloguing the fields that would not', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Data Science & Analytics
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Specialized
Rationale: Auditing whether features existed at prediction time is a production machine-learning response to data leakage.
Related originating lineages:
- Computer Science & Software Engineering — Data-pipeline and serving semantics shape whether a feature actually exists at scoring time.
- Statistics & Experimental Design — Temporal ordering and prospective-design discipline provide the inferential basis for the audit.
Review resolution: Both reviewers agree that data_science is primary. I retain statistics_experimental_design, computer_science only as formative origin lineage(s), without treating every later application as an origin. cross_disciplinary_synthesis is appropriate because the exact artifact combines contributions from multiple professional lineages. Reach is specialized as a separate applicability judgment: it does not widen or narrow the recorded provenance. Encyclopedia synthesis is false because the artifact is already established enough that encyclopedia-specific synthesis is not required. The secondary differences are reconciled with no unresolved primary-provenance ambiguity.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
A feature that fails the audit cannot be rescued by a cleverer split. This is the audit's whole reason to exist as a separate, upfront step: temporal holdouts, entity grouping, and duplicate scans all assume the feature set is legitimate and only the partition is at risk. If a post-outcome column survives into the feature set, every one of those defences will still report a leaked result.
References¶
[1] Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. "Leakage in Data Mining: Formulation, Detection, and Avoidance". ACM Transactions on Knowledge Discovery from Data 6(4), Article 15, 1–21 (2012). Defines target leakage as illegitimate target information and combines formal review with methods for detecting leakage in observed data. registry ↩