Linear Discriminant Analysis¶

A supervised linear projection and classifier that separates labeled classes relative to their within-class covariance.

Version: v2 · 2026-09-19 · History
Domain-specific #: 2186
Domain group: Formal Sciences
Origin domain: Experimental Design & Statistics
Subdomain: Multivariate Analysis → Experimental Design & Statistics
Aliases: LDA, Fisher discriminant analysis, Discriminant function analysis

Core Idea¶

Linear discriminant analysis (LDA) is a supervised multivariate method that finds linear combinations of features that separate labeled classes relative to variation within those classes. In the two-class Fisher formulation, a direction \(w\) maximizes

\[ J(w)=\frac{w^TS_Bw}{w^TS_Ww}, \]

where \(S_B\) represents between-class separation and \(S_W\) represents pooled within-class scatter. The solution is proportional to \(S_W^{-1}(\mu_1-\mu_0)\) when the inverse exists. Fisher introduced the discriminant criterion in his analysis of multiple measurements.

In a probabilistic classification formulation, class-conditional multivariate normal distributions share a covariance matrix but have different means and priors. Log-posterior comparisons then produce linear decision boundaries. In multiclass settings, generalized eigenvectors of between- and within-class scatter produce at most \(K-1\) discriminant coordinates for \(K\) classes.

How would you explain it like I'm…

The Best Sorting Score

Imagine a pile of apples and pears that are already labeled. You want one way of measuring them that puts the apples on one side and the pears on the other. The best way is one where the two groups end up far apart while each group stays bunched close together. Linear Discriminant Analysis finds that best way.

Far Apart, Tight Together

Linear Discriminant Analysis is a way to tell groups apart using measurements, like height and weight. You start with examples that are already labeled. Then you find one combined score, mixing the measurements in the best amounts, so the groups end up far apart while each group stays tightly packed. It is not just any line through the data: it cares about both the gap between groups and how spread out each group is. New items can be sorted by their score.

Fisher's Class-Separating Direction

Linear Discriminant Analysis (LDA) is a supervised method: it needs labeled classes. It searches for a weighted combination of features (a direction w) that maximizes the ratio of between-class separation to within-class spread. Fisher's version for two classes picks w maximizing (wᵀS_B w)/(wᵀS_W w), where S_B measures how far apart the class means are and S_W measures pooled scatter inside classes; the answer points along S_W⁻¹(μ₁ − μ₀). Unlike PCA, which ignores labels and just finds directions of largest variance, LDA uses the labels to find directions that separate groups. A probabilistic version assumes each class is a normal distribution with its own mean but a shared covariance, which yields straight-line decision boundaries. With K classes, it produces at most K−1 useful discriminant directions.

Linear Discriminant Analysis is a supervised multivariate technique that finds linear combinations of features separating labeled classes relative to their within-class variation. Fisher's two-class criterion maximizes J(w) = wᵀS_B w / wᵀS_W w, where S_B is between-class scatter and S_W is pooled within-class scatter; the maximizer is proportional to S_W⁻¹(μ₁ − μ₀) when S_W is invertible. In the probabilistic formulation, class-conditional densities are multivariate normal with distinct means and priors but a common covariance matrix, so the quadratic terms cancel in log-posterior comparisons and decision boundaries are linear. For K classes, generalized eigenvectors of the pair (S_B, S_W) give at most K−1 discriminant coordinates, used for dimensionality reduction or classification. The defining ingredients are labels, shared within-class geometry, a linear score space, and a classification or reduction purpose. Merely projecting data or fitting some linear separator does not count as LDA.

Scope of Application¶

LDA is used for supervised classification, feature extraction, visualization of class structure, and preprocessing. It is appropriate when linear combinations are meaningful and pooled covariance can be estimated reliably. Classical morphometrics, pattern recognition, spectroscopy, and moderate-dimensional labeled data are common settings.

When dimension approaches or exceeds sample size, \(S_W\) can be singular. Regularized or shrinkage variants may then replace the inverse. Strongly unequal covariances, nonlinear class manifolds, outliers, or severe nonnormality can make another model preferable. The Fisher projection criterion itself can still be used descriptively without claiming a full Gaussian data-generating model; that distinction must be stated.

Clarity¶

LDA clarifies which variation matters. A direction with large overall variance may be useless if all classes vary together. A direction with modest overall variance can be discriminative when class means differ and within-class spread is small. The ratio criterion makes that tradeoff explicit.

It also separates score construction from decision policy. Projected coordinates represent discriminant axes. Classification additionally uses priors, centroids, or discriminant functions.

Manages Complexity¶

For \(p\) original variables and \(K\) classes, LDA can compress discriminative information into at most \(K-1\) axes. Analysts can visualize groups, reduce subsequent model dimension, and identify combinations of measurements that carry separation. The pooled covariance shares information across classes instead of estimating a full covariance for each.

Abstract Reasoning¶

The Rayleigh quotient implies scale invariance: multiplying \(w\) by a nonzero scalar does not change \(J(w)\). Optimization yields a generalized eigenproblem \(S_Bw=\lambda S_Ww\). Eigenvalues quantify separation along corresponding discriminant directions, while eigenvectors can be chosen \(S_W\)-orthogonal under suitable conditions.

The Gaussian classifier gives another derivation. Expanding log densities cancels the common quadratic term, leaving a linear function of \(x\).

Knowledge Transfer¶

The same within-versus-between geometry transfers across labeled measurement domains. A direction learned in one dataset does not transfer numerically unless variables, scaling, populations, and labels align, but the criterion and validation procedure do.

The portable parent is Dimensionality Reduction: map high-dimensional observations to a smaller coordinate system. LDA adds supervision and covariance-weighted class separation. Calling any “discriminant” narrative LDA without feature vectors and labels is metaphorical.

Relationships to Other Abstractions¶

Current abstraction Linear Discriminant Analysis Domain-specific

Parents (1) — outgoing typed relations

Linear Discriminant Analysis is a kind of Dimensionality Reduction Prime

LDA specializes Dimensionality Reduction by preserving supervised class separation.

Hierarchy paths (4) — routes to 3 parentless roots

Linear Discriminant Analysis → is a kind of Dimensionality Reduction → is a kind of Approximation → is a decomposition of Representation → presupposes Abstraction

Show alternative paths (3)

Neighborhood in Abstraction Space¶

Linear Discriminant Analysis sits in a sparse region of the domain-specific corpus (88^th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Unclustered & Miscellaneous (2769 abstractions)

Nearest neighbors

Violin Plot — 0.83
Boosting — 0.83
Gaussian Naive Bayes — 0.80
Quantification (machine learning) — 0.80
Information Gain for Decision-Tree Splits — 0.79

Computed from structural-signature embeddings · 2026-10-25