Linear Discriminant Analysis¶

A supervised linear projection and classifier that separates labeled classes relative to their within-class covariance.

Version: v1 · 2026-08-30 · History
Domain-specific #: 2186
Origin domain: statistics
Subdomain: multivariate analysis
Aliases: LDA, Fisher discriminant analysis, Discriminant function analysis

Core Idea¶

Linear discriminant analysis (LDA) is a supervised multivariate method that finds linear combinations of features that separate labeled classes relative to variation within those classes. In the two-class Fisher formulation, a direction \(w\) maximizes

\[ J(w)=\frac{w^TS_Bw}{w^TS_Ww}, \]

where \(S_B\) represents between-class separation and \(S_W\) represents pooled within-class scatter. The solution is proportional to \(S_W^{-1}(\mu_1-\mu_0)\) when the inverse exists. Fisher introduced the discriminant criterion in his analysis of multiple measurements.^[1]

In a probabilistic classification formulation, class-conditional multivariate normal distributions share a covariance matrix but have different means and priors. Log-posterior comparisons then produce linear decision boundaries. In multiclass settings, generalized eigenvectors of between- and within-class scatter produce at most \(K-1\) discriminant coordinates for \(K\) classes.^[2]

The abstraction includes labeled supervision, shared within-class geometry, a linear score space, and either classification or dimension-reduction use. It is not merely projecting data or drawing a linear separator.

Structural Signature¶

Mandatory roles:

Feature vectors \(x\in\mathbb R^p\) provide a common measurement space.
Known class labels define supervised groups.
Class means \(\mu_k\) locate groups.
Within-class scatter or pooled covariance \(S_W\) defines noise geometry.
Between-class scatter \(S_B\) defines separation geometry.
Discriminant vectors solve a generalized eigenproblem or equivalent criterion.
Projected scores or linear discriminant functions support classification and/or reduction.

Recognition test. A method qualifies when labeled class separation is optimized relative to pooled within-class variation and produces linear discriminant scores. A generic supervised linear model or unsupervised projection is insufficient.

What It Is Not¶

It is not principal component analysis, which ignores labels and maximizes total variance rather than class separation.
It is not logistic regression; logistic regression models conditional class probabilities directly without the same Gaussian/shared-covariance generative assumption.
It is not quadratic discriminant analysis, which allows class-specific covariances and quadratic boundaries.
It is not latent Dirichlet allocation, which shares the acronym LDA but models topic mixtures.
It is not any linear classifier. Support-vector machines and perceptrons optimize different criteria.

Scope of Application¶

LDA is used for supervised classification, feature extraction, visualization of class structure, and preprocessing. It is appropriate when linear combinations are meaningful and pooled covariance can be estimated reliably. Classical morphometrics, pattern recognition, spectroscopy, and moderate-dimensional labeled data are common settings.

When dimension approaches or exceeds sample size, \(S_W\) can be singular. Regularized or shrinkage variants may then replace the inverse. Strongly unequal covariances, nonlinear class manifolds, outliers, or severe nonnormality can make another model preferable. The Fisher projection criterion itself can still be used descriptively without claiming a full Gaussian data-generating model; that distinction must be stated.

Preprocessing is part of the statistical specification. Units and redundant variables affect the pooled covariance, while data leakage can make supervised separation look implausibly strong. Any scaling, imputation, feature selection, or regularization must be learned inside the training split. A discriminant plot made after using all labels to select features is descriptive unless a properly nested validation reproduces it.

Clarity¶

LDA clarifies which variation matters. A direction with large overall variance may be useless if all classes vary together. A direction with modest overall variance can be discriminative when class means differ and within-class spread is small. The ratio criterion makes that tradeoff explicit.

It also separates score construction from decision policy. Projected coordinates represent discriminant axes. Classification additionally uses priors, centroids, or discriminant functions. Changing class priors can shift decision thresholds without changing the within/between scatter directions.

Manages Complexity¶

For \(p\) original variables and \(K\) classes, LDA can compress discriminative information into at most \(K-1\) axes. Analysts can visualize groups, reduce subsequent model dimension, and identify combinations of measurements that carry separation. The pooled covariance shares information across classes instead of estimating a full covariance for each.

Compression can conceal subgroup heterogeneity or nonlinear structure. It does not solve label error, missing data, or class imbalance by itself. A good training projection may overfit if covariance is unstable, so validation remains necessary.

The low-dimensional scores also make diagnostics manageable. Analysts can inspect class centroids, score distributions, influential observations, and covariance fit in the discriminant space. These summaries do not replace examination in the original features, but they focus attention on the directions the model actually uses.

Abstract Reasoning¶

The Rayleigh quotient implies scale invariance: multiplying \(w\) by a nonzero scalar does not change \(J(w)\). Optimization yields a generalized eigenproblem \(S_Bw=\lambda S_Ww\). Eigenvalues quantify separation along corresponding discriminant directions, while eigenvectors can be chosen \(S_W\)-orthogonal under suitable conditions.

The Gaussian classifier gives another derivation. Expanding log densities cancels the common quadratic term, leaving a linear function of \(x\). If class covariances differ, cancellation fails and the decision boundary becomes quadratic. This is a structural boundary, not cosmetic terminology.

Affine changes of feature coordinates transform the discriminant vectors while preserving the represented separating subspace under appropriate nonsingular transformations. This explains why raw coefficient magnitudes cannot be compared across different measurement units without accounting for scaling. The invariant object is the score/subspace with its covariance geometry, not a bare coefficient list.

Knowledge Transfer¶

The same within-versus-between geometry transfers across labeled measurement domains. A direction learned in one dataset does not transfer numerically unless variables, scaling, populations, and labels align, but the criterion and validation procedure do.

The portable parent is Dimensionality Reduction: map high-dimensional observations to a smaller coordinate system. LDA adds supervision and covariance-weighted class separation. Calling any “discriminant” narrative LDA without feature vectors and labels is metaphorical.

Examples¶

Two classes. Suppose two groups have means \(\mu_0\) and \(\mu_1\) with pooled covariance \(\Sigma\). The direction \(w=\Sigma^{-1}(\mu_1-\mu_0)\) downweights noisy correlated directions and upweights stable mean differences. Projecting \(x\) to \(w^Tx\) yields a one-dimensional score.

Three classes. With three labeled species and several measurements, at most two discriminant axes carry mean-separation information. Plotting them can reveal whether one species separates on the first axis and the others on the second. Cross-validation is needed before treating visual separation as predictive performance.

Boundary. PCA may select a high-variance nuisance measurement shared by all classes. LDA can ignore it if it contributes little between-class scatter. Conversely, a mislabeled sample can strongly distort class means and covariance.

Structural Tensions¶

Separation versus within-class stability: moving means apart is useful only relative to scatter. Diagnostic: are both numerator and denominator of the discriminant criterion reported?
Dimension reduction versus information loss: few axes simplify modeling but may omit within-class structure. Diagnostic: do held-out results remain stable as axes are added?
Pooled covariance versus heterogeneous classes: sharing improves estimation but can misrepresent unequal groups. Diagnostic: do class-specific covariance diagnostics indicate material differences?
Closed-form clarity versus singularity: the inverse formula is simple but may not exist in high dimensions. Diagnostic: what rank, regularization, or shrinkage treatment is used?
Descriptive projection versus generative classifier: Fisher’s criterion need not assert Gaussianity. Diagnostic: is the claim about visualization/separation or calibrated class probabilities?

Structural–Framed Character¶

LDA is strongly structural within statistics. Its scatter matrices, eigenproblem, and linear scores are mathematical. Class definitions and measurement choices frame an application, but do not alter the method’s identity. The method supports exact computational checks and falsifiable assumptions.

It remains domain-specific because labeled vector data and covariance geometry are indispensable. The cross-domain abstraction is dimensionality reduction or classification.

Structural Core vs. Domain Accent¶

Structural core. Reduce a representation while preserving a task-relevant distinction, balancing signal between groups against noise within groups.

Domain accent. Feature vectors, labeled classes, means, pooled covariance, scatter matrices, generalized eigenvectors, and linear decision functions define LDA. Removing them leaves generic supervised reduction.

The remaining domain structure is autonomous because it determines formulas, rank limits, assumptions, and boundary cases.

LDA specializes Dimensionality Reduction by preserving supervised class separation. It relates to Classification, Optimization, and Signal Extraction. These parents do not entail the Fisher ratio or shared-covariance decision geometry. Dimensionality Reduction is the minimal parent for the accepted node.

Relationships to Other Abstractions¶

Current abstraction Linear Discriminant Analysis Domain-specific

Parents (1) — more general patterns this builds on

Linear Discriminant Analysis is a kind of Dimensionality Reduction Prime

LDA specializes Dimensionality Reduction by preserving supervised class separation.

Hierarchy paths (4) — routes to 3 parentless roots

Linear Discriminant Analysis → Dimensionality Reduction → Approximation → Representation → Abstraction

Show alternative paths (3)

Neighborhood in Abstraction Space¶

Linear Discriminant Analysis sits in a sparse region of the domain-specific corpus (86^th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Unclustered & Miscellaneous (1565 abstractions)

Nearest neighbors

Violin Plot — 0.83
Boosting — 0.83
Quantification (machine learning) — 0.80
Polynomial Chaos Expansion — 0.79
AdaBoost — 0.78

Computed from structural-signature embeddings · 2026-09-08

Not to Be Confused With¶

PCA: unsupervised total-variance projection. Tell: are labels used in the objective?
Quadratic discriminant analysis: class-specific covariances. Tell: are decision boundaries linear or quadratic?
Logistic regression: conditional likelihood classifier. Tell: is a pooled Gaussian covariance model or Fisher scatter criterion central?
Latent Dirichlet allocation: topic model with the same acronym. Tell: are observations documents and topics rather than class-labeled feature vectors?
Canonical correlation analysis: maximizes correlation between paired variable sets. Tell: is the target class separation or cross-view association?

References¶

[1] R. A. Fisher, “The Use of Multiple Measurements in Taxonomic Problems,” Annals of Eugenics 7.2 (1936), 179–188, https://doi.org/10.1111/j.1469-1809.1936.tb02137.x. registry ↩

[2] Trevor Hastie, Robert Tibshirani, and Jerome Friedman, The Elements of Statistical Learning, 2^nd ed., Springer, 2009, §4.3, https://doi.org/10.1007/978-0-387-84858-7. registry ↩