PRESS Statistic¶

The sum of squared leave-one-out prediction errors from a fitted regression model, computed by refitting without each case or through leverage-adjusted ordinary residuals.

Version: v1 · 2026-08-30 · History
Domain-specific #: 2523
Origin domain: statistics
Subdomain: regression diagnostics and cross-validation
Aliases: Prediction sum of squares, Predicted residual sum of squares, PRESS

Core Idea¶

The prediction error sum of squares (PRESS) evaluates an ordinary least-squares regression by withholding each observation in turn, predicting it from the model fitted to the remaining observations, squaring the deleted residual, and summing across cases. It therefore measures within-sample leave-one-out predictive discrepancy rather than training fit.^[1]

For a fixed full-rank linear model fitted by OLS, the deleted residual can be computed without n refits as e_i/(1-h_ii), where e_i is the ordinary residual and h_ii is leverage. Thus PRESS=Σ[e_i/(1-h_ii)]². High-leverage cases are amplified because removing them changes the fit substantially. The identity depends on the fixed design/model; feature selection, preprocessing, tuning, dependence, weights, regularization, and clustered data require the entire modeling procedure to be replayed or a different validation design.

Structural Signature¶

The dataset and outcome. Cases, predictors, and target are fixed.
The modeling procedure. Formula, preprocessing, and fitting rule are declared.
The leave-one-out split. Each case is excluded once from training.
The deleted prediction. The held-out outcome is predicted without using itself.
The deleted residual. Observed minus held-out prediction records error.
The squared aggregation. Deleted errors are squared and summed.
The leverage shortcut. Under OLS assumptions, diagonal hat values reproduce refit residuals.
The comparator. PRESS is interpreted relative to outcome scale, alternative models, or a baseline.
The dependence boundary. Exchangeability and procedure nesting determine whether validation is meaningful.

What It Is Not¶

Not the training residual sum of squares. Each PRESS prediction excludes its own outcome.
Not a genuinely external test set. All cases contribute to training in other folds.
Not automatically unbiased for every deployment setting. Dependence and distribution shift matter.
Not valid after selection done once on the full data. Selection must be nested in each fold.
Not simply residual standard error. It is an unnormalized predictive sum.
Not robust to leverage by definition. Leverage can dominate the statistic.

Scope of Application¶

PRESS is literal in linear-regression diagnostics and related small-sample predictive comparisons where leave-one-out validation is suitable.

Regression comparison. Contrasting fixed candidate predictor sets.
Influence diagnosis. Finding cases with large leverage-adjusted deleted residuals.
Chemometrics. Evaluating calibration models under carefully nested procedures.
Response-surface modeling. Checking predictive adequacy inside a design region.
Teaching. Connecting hat-matrix geometry with cross-validation.
Approximate validation. Computing exact OLS leave-one-out error efficiently.
Sensitivity analysis. Comparing PRESS with training error and external validation.

Clarity¶

Report outcome scale, cases, model matrix, intercept, weights, preprocessing, missing-data rule, feature selection/tuning, leverage values, deleted residuals, aggregation, normalization if any, and dependence structure. State whether the analytic shortcut or explicit refits were used. Compare models only on identical outcomes and folds.

Declare the fitted model, response scale, observation unit, and whether deletion is truly one case at a time. For ordinary least squares with a fixed design, the deleted residual can be computed from the ordinary residual and leverage, avoiding repeated refits; that identity should not be exported unchanged to every penalized, nonlinear, or adaptive model. PRESS is the sum of squared deleted prediction errors, while a mean version differs by a denominator and predicted R-squared adds another reference quantity. High-leverage observations can magnify deleted residuals because their fitted values depend strongly on their own presence. Model selection performed before or within cross-validation changes the target: leaving one point out after selecting variables on the full data leaks information. Dependence, groups, repeated measures, and time order can make casewise deletion an inappropriate prediction design.

Manages Complexity¶

The leverage identity compresses n OLS refits into one fit and exposes influential observations. A single sum supports model comparison. Aggregation can hide concentrated failure, and the shortcut ceases to represent the actual pipeline when transformations or selections depend on all observations.

Training residuals are optimistically small because each observation helps fit the model that predicts it. PRESS removes that direct self-influence by predicting every case from the model fitted without it, then aggregates the squared discrepancies. The leverage identity turns an apparently expensive sequence of refits into a diagnostic computed from one ordinary least-squares fit. That compression also explains behavior: a modest ordinary residual at high leverage can become a large deleted residual. PRESS supports comparison of candidate models on the same observations, but repeated selection can still overfit the validation criterion and correlated data can make withheld cases too easy. The abstraction manages complexity by combining local influence and predictive discrepancy in one sum, not by creating an external test set.

Abstract Reasoning¶

Freeze the predictive procedure and dataset.
Exclude one observation.
Fit every data-dependent step on the remaining cases.
Predict the held-out outcome.
Record and square the deleted residual.
Repeat for all observations.
Sum or transparently normalize errors.
Inspect case contributions and validate deployment relevance.
The central derivation uses the geometry of a linear smoother. An ordinary fitted value contains a fraction of its own observed response measured by leverage. Deleting that observation removes the self-contribution and changes the fitted coefficients; algebra expresses the resulting held-out discrepancy as the ordinary residual divided by one minus leverage. Squaring and summing those deleted discrepancies yields PRESS without fitting n separate models. The denominator explains the statistic's sensitivity near unit leverage and is a diagnostic, not a numerical inconvenience to suppress. When the estimator is not ordinary least squares with a fixed design, the shortcut must be re-derived or replaced by actual refitting. Sound model comparison also holds the response, cases, loss, and selection protocol fixed. Thus PRESS reasoning combines projection geometry, influence, and prediction while avoiding the false inference that a low value proves external generalization.

Knowledge Transfer¶

PRESS is residual analysis turned outward: each discrepancy is computed against a prediction that did not train on that observation. Residual Analysis is the strict parent; leave-one-out refitting and the leverage identity supply the statistical accent.

Residual Analysis is the strict parent because PRESS transforms fitted errors into leave-one-out residuals and aggregates them to diagnose predictive behavior. The transferable pattern is remove each case's fitting influence → predict it → aggregate out-of-fit discrepancy. It relates to cross-validation generally, but the PRESS identity and leverage adjustment belong specifically to ordinary least-squares geometry. Residual sum of squares measures in-sample fit; Cook's distance measures parameter influence; neither is the same output. The domain residual includes deleted residuals, the hat matrix, leverage, squared-error aggregation, and fixed-design assumptions.

Examples¶

Canonical¶

In fixed OLS, an observation with ordinary residual e_i and leverage h_ii has deleted residual e_i/(1-h_ii). Summing its square across observations produces exactly the explicit leave-one-out PRESS value.^[1]

Mapped back: training residual + leverage correction → deleted prediction residual → aggregate leave-one-out error.

Applied / In Practice¶

A small PRESS improvement disappears when predictor selection is rerun inside each leave-one-out fold. The original result leaked held-out outcomes through a full-data selection step and did not validate the claimed procedure.

Two linear models have similar residual sums of squares. The second includes a flexible predictor that fits one high-leverage case closely. Its ordinary residual is small, but deleting that case changes the fitted surface and produces a large deleted residual, raising PRESS. The analyst compares models using the same observations and predictor-selection protocol, inspects the influential case, and reports the response-unit scale. When observations are grouped by subject, the analyst switches to groupwise validation rather than claiming casewise PRESS estimates new-subject performance. The example shows both the leverage mechanism and the transfer boundary.

Mapped back: full-pipeline nesting audit → leakage removal → corrected predictive comparison.

Structural Tensions¶

Training fit vs. predictive fit. A flexible model lowers RSS but can worsen deleted predictions. Diagnostic: Does PRESS improve against a fixed baseline?
Efficient shortcut vs. pipeline complexity. Hat-matrix algebra covers fixed OLS, not arbitrary workflows. Diagnostic: Which steps change when a case is removed?
Aggregate score vs. influential cases. One case can dominate. Diagnostic: What are the per-case contributions?
Leave-one-out reuse vs. dependence. Highly related cases leak information across folds. Diagnostic: Should groups or time blocks be held out?
Autonomous statistic vs. generic residual analysis. Many residual sums exist; delete-one prediction defines PRESS. Diagnostic: Was each outcome excluded from its predictor fit?

Structural–Framed Character¶

PRESS is structural under a declared dataset and model. Computation is objective; the procedure, folds, and deployment analogy frame interpretation. It is evaluatively neutral. Residual Analysis supplies error decomposition; cross-validation changes the information boundary.

Case deletion, refit-or-equivalent update, held-out prediction, squared deleted residual, and aggregation are structural. Dataset, response units, predictor coding, variable-selection procedure, and whether the sum or mean is reported are framed. The shortcut formula is structural only under the model class that justifies it. Scaling the response rescales PRESS, so comparisons require consistent outcomes or normalization. This framing protects the statistic from being presented as a universal scale-free model score.

Structural Core vs. Domain Accent¶

The skeleton is prediction + excluded evidence → residual → aggregate discrepancy. The accent is regression, leave-one-out folds, squared deleted residuals, hat leverage, and predictive model comparison. Remove those and one has residual analysis generally.

The portable core is measure prediction after removing the target's direct contribution to fitting. The regression accent is ordinary least squares, the hat matrix, leverage-adjusted residuals, and a sum of squared errors. Remove deletion and the result collapses to training residual analysis. Keep leave-one-out prediction but change loss or estimator and the procedure is cross-validation rather than necessarily the PRESS statistic under its classical identity. The residual is both predictive and influence-aware.

Residual Analysis is the strict parent because PRESS aggregates observed-minus-predicted discrepancies; it narrows the parent by requiring each prediction to exclude the observed case.

The prospective workspace queue contains one strict upward edge to prime:residual_analysis. No live DAG mutation is authorized.

Relationships to Other Abstractions¶

Current abstraction PRESS Statistic Domain-specific

Parents (1) — more general patterns this builds on

PRESS Statistic is a kind of Aggregation Prime

The accepted reference-grade review places PRESS Statistic under Aggregation because the child instantiates or depends on the parent's broader structure while retaining its own constitutive identity.

Hierarchy path (1) — routes to 1 parentless root

PRESS Statistic → Aggregation → Micro Macro Linkage

Neighborhood in Abstraction Space¶

PRESS Statistic sits in a sparse region of the domain-specific corpus (88^th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Unclustered & Miscellaneous (1565 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-09-08

Not to Be Confused With¶

Residual sum of squares. In-sample training error.
Leave-one-out cross-validation. The broader resampling design; PRESS is its squared-error aggregate for regression.
Predicted R-squared. A normalization derived from PRESS under a chosen baseline.
Cook's distance. An influence measure for coefficient/fit change.
Externally studentized residual. A scale-adjusted deleted residual.
Test-set error. Evaluation on data never used in any fitting or selection.

References¶

[1] David M. Allen, ‘The Relationship between Variable Selection and Data Augmentation and a Method for Prediction,’ Technometrics 16, no. 1 (1974): 125–127, https://doi.org/10.1080/00401706.1974.10489157. registry ↩a ↩b