Preprocessing Fit-on-Training-Only¶
Pipeline hygiene rule — instantiates Leakage-Resistant Validation Design
Requires every fitted transform — scalers, imputers, encoders, vectorizers, feature selectors, resamplers — to learn its parameters from the training partition alone, then apply unchanged to validation and test.
Preprocessing feels innocent, which is exactly why it leaks. A scaler that centers on the whole dataset's mean, an imputer that fills with a global median, a vectorizer whose vocabulary is built from every document, a feature selector ranked on all rows — each has quietly read the test set and folded a little of it into the model. Preprocessing Fit-on-Training-Only is the rule that forbids this: any transformation with learned parameters must estimate them on the training partition only, and then be applied — frozen — to validation and test. Its defining move is to police the fitting step of the pipeline, the one place where a leak enters through a statistic rather than a feature or a join, and to insist the split come before any transform is fit, not after.
Example¶
An NLP team builds a spam classifier. The convenient pipeline vectorizes the entire corpus with TF-IDF, runs a chi-squared feature selection over all documents, and then splits into train and test. Test accuracy reads a delighted ≈97%. The catch: the vocabulary and the IDF weights were computed over the test documents too, and the selected features were ranked using test labels — the model was handed a representation shaped by the very data it was about to be graded on. Re-run under the fit-on-training-only rule — split first, fit the vectorizer and the selector on the training fold, transform the test fold with those frozen parameters — and accuracy settles to ≈93%. The missing four points were preprocessing leakage. The rule also stamps each transform with a provenance note recording which partition its parameters came from, so the discipline is auditable rather than assumed.
How it works¶
What distinguishes it from ordinary preprocessing is a strict ordering and a fit/apply split enforced on every stateful transform:
- Split first. The train/validation/test partition is drawn before any transform sees the data, so no fitting step can span the boundary.
- Fit on train, transform elsewhere. Each transform's parameters — means, medians, category maps, vocabularies, selected-feature lists, resampling — are estimated on the training partition and then applied unchanged downstream.
- Inside every fold. Under cross-validation the fit/apply cycle repeats within each fold, never once over the pooled data.
- Record provenance. Each fitted transform carries which partition (and, under CV, which fold) it learned from, making the isolation checkable.
Tuning parameters¶
- Transform inventory — which steps count as "fitted" and must obey the rule. Miss one stateful step — a target encoder, an outlier bound, an oversampler — and the leak persists through it; err toward treating every parameter-learning step as in-scope.
- Fold-level vs. dataset-level fitting — whether transforms refit inside each CV fold or once on a fixed training set. Per-fold is stricter and costlier; dataset-level is fine only against a single frozen holdout.
- Unseen-category / out-of-range handling — how the frozen transform treats values it never saw in training (a new category, an out-of-range number). The policy must be decided in advance, because reacting to test values is itself a leak.
- Resampling placement — whether class balancing (e.g. oversampling) happens strictly inside the training fold. Balancing before the split is a common and severe leak.
When it helps, and when it misleads¶
Its strength is that it seals the most underestimated leak: teams who would never join a future column think nothing of scaling before splitting, and the resulting inflation is small enough to be believed and large enough to matter. Encoding the rule in a pipeline abstraction makes correct behavior the default and the leak hard to reintroduce by accident.
Its failure mode is coverage: the rule only protects the transforms it is applied to, and a single stateful step invoked outside the managed pipeline — a quick global normalization in a notebook, a lookup table built once over everything — reopens the boundary invisibly. The classic misuse is to satisfy the rule mechanically while defeating it in spirit, for instance by fitting on train but choosing the transform's design (which features to keep, how many bins) by peeking at test performance. The discipline that guards against it is to route all preprocessing through one enforced pipeline whose transforms fit only within the training partition,[n1] and to make design choices about preprocessing under the same isolation as the parameters.
How it implements the components¶
Preprocessing Fit-on-Training-Only realizes the transform-isolation side of the archetype — sealing the fitting step, not selecting models or timing data:
pipeline_isolation_rule— it is the rule that no held-out data enters any fitting step: transforms learn on training data and are applied frozen elsewhere.feature_provenance_record— each transform is stamped with the partition (and fold) its parameters were estimated from, making the isolation auditable rather than assumed.
It isolates transform fitting; isolating model selection from the reported score is Nested Cross-Validation, and the temporal provenance of source values — joining only as-of-decision-time data — is As-Of Join Rule.
Related¶
- Instantiates: Leakage-Resistant Validation Design — this rule seals the preprocessing stage, where a leak enters as a statistic rather than a feature.
- Consumes: the split it fits within is defined by Entity-Grouped Split, Time-Based Holdout, or the folds of Nested Cross-Validation.
- Sibling mechanisms: Nested Cross-Validation · As-Of Join Rule · Entity-Grouped Split · Time-Based Holdout · Feature Availability Audit · Label Proxy Screen · Leakage Ablation Test · Duplicate and Near-Duplicate Scan · Benchmark Deduplication Scan · Fresh Holdout Retest · Holdout Access Log
Editorial Notes¶
Form Classification¶
Form family: Rule, Policy & Commitment
Rationale: Preprocessing Fit-on-Training-Only operates as a standing rule, threshold, contractual commitment, or policy constraint governing future conduct because it requires every fitted transform — scalers, imputers, encoders, vectorizers, feature selectors, resamplers — to learn its parameters from the training partition alone, then apply unchanged to validation and test.
Independent corroboration: The frozen evidence defines Preprocessing Fit-on-Training-Only as 'Requires every fitted transform — scalers, imputers, encoders, vectorizers, feature selectors, resamplers — to learn its parameters from the training partition alone, then apply unchanged to validation and test', so its operative form is Rule, Policy & Commitment.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Data Science & Analytics
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Specialized
Rationale: Preprocessing Fit-on-Training-Only is most plausibly rooted in the data_science tradition because its characteristic form depends on production data pipelines, predictive modeling, and machine-learning validation. The assignment tracks that formative lineage, not the many settings in which the mechanism can now be applied.
Related originating lineages:
- Computer Science & Software Engineering — The computer_science tradition materially shaped Preprocessing Fit-on-Training-Only through its own practice of algorithms, data structures, formal interfaces, and software-system practice.
- Statistics & Experimental Design — The statistics_experimental_design tradition materially shaped Preprocessing Fit-on-Training-Only through its own practice of probability, calibrated inference, experimental design, and uncertainty analysis.
Review outcome: Independent reviewer agreement; high confidence.
Notes¶
The rule and the split are separate but co-dependent: fit-on-training-only is meaningless until some mechanism has drawn the training boundary, and any split is undermined the moment a transform is fit across it. Whenever the split changes — new folds, a new time cutoff — every fitted transform must be re-fit within the new training partition, not carried over.
[n1] Preprocessing leakage — fitting any transformation (scaling, imputation, encoding, vectorization, feature selection, resampling) on data that includes the evaluation set lets test-set statistics influence the model and inflates measured performance. The standard guard is to fit transforms inside the training partition only, as enforced by pipeline abstractions such as scikit-learn's Pipeline. ↩