Linear Discriminant Analysis¶
A supervised linear projection and classifier that separates labeled classes relative to their within-class covariance.
Core Idea¶
Linear discriminant analysis (LDA) is a supervised multivariate method that finds linear combinations of features that separate labeled classes relative to variation within those classes. In the two-class Fisher formulation, a direction \(w\) maximizes
where \(S_B\) represents between-class separation and \(S_W\) represents pooled within-class scatter. The solution is proportional to \(S_W^{-1}(\mu_1-\mu_0)\) when the inverse exists. Fisher introduced the discriminant criterion in his analysis of multiple measurements.
In a probabilistic classification formulation, class-conditional multivariate normal distributions share a covariance matrix but have different means and priors. Log-posterior comparisons then produce linear decision boundaries. In multiclass settings, generalized eigenvectors of between- and within-class scatter produce at most \(K-1\) discriminant coordinates for \(K\) classes.
Scope of Application¶
LDA is used for supervised classification, feature extraction, visualization of class structure, and preprocessing. It is appropriate when linear combinations are meaningful and pooled covariance can be estimated reliably. Classical morphometrics, pattern recognition, spectroscopy, and moderate-dimensional labeled data are common settings.
When dimension approaches or exceeds sample size, \(S_W\) can be singular. Regularized or shrinkage variants may then replace the inverse. Strongly unequal covariances, nonlinear class manifolds, outliers, or severe nonnormality can make another model preferable. The Fisher projection criterion itself can still be used descriptively without claiming a full Gaussian data-generating model; that distinction must be stated.
Clarity¶
LDA clarifies which variation matters. A direction with large overall variance may be useless if all classes vary together. A direction with modest overall variance can be discriminative when class means differ and within-class spread is small. The ratio criterion makes that tradeoff explicit.
It also separates score construction from decision policy. Projected coordinates represent discriminant axes. Classification additionally uses priors, centroids, or discriminant functions.
Manages Complexity¶
For \(p\) original variables and \(K\) classes, LDA can compress discriminative information into at most \(K-1\) axes. Analysts can visualize groups, reduce subsequent model dimension, and identify combinations of measurements that carry separation. The pooled covariance shares information across classes instead of estimating a full covariance for each.
Abstract Reasoning¶
The Rayleigh quotient implies scale invariance: multiplying \(w\) by a nonzero scalar does not change \(J(w)\). Optimization yields a generalized eigenproblem \(S_Bw=\lambda S_Ww\). Eigenvalues quantify separation along corresponding discriminant directions, while eigenvectors can be chosen \(S_W\)-orthogonal under suitable conditions.
The Gaussian classifier gives another derivation. Expanding log densities cancels the common quadratic term, leaving a linear function of \(x\).
Knowledge Transfer¶
The same within-versus-between geometry transfers across labeled measurement domains. A direction learned in one dataset does not transfer numerically unless variables, scaling, populations, and labels align, but the criterion and validation procedure do.
The portable parent is Dimensionality Reduction: map high-dimensional observations to a smaller coordinate system. LDA adds supervision and covariance-weighted class separation. Calling any “discriminant” narrative LDA without feature vectors and labels is metaphorical.
Relationships to Other Abstractions¶
Current abstraction Linear Discriminant Analysis Domain-specific
Parents (1) — more general patterns this builds on
-
Linear Discriminant Analysis is a kind of Dimensionality Reduction Prime
LDA specializes Dimensionality Reduction by preserving supervised class separation.
Hierarchy paths (4) — routes to 3 parentless roots
- Linear Discriminant Analysis → Dimensionality Reduction → Approximation → Representation → Abstraction
- Linear Discriminant Analysis → Dimensionality Reduction → Compression → Abstraction
- Linear Discriminant Analysis → Dimensionality Reduction → Compression → Optimization
- Linear Discriminant Analysis → Dimensionality Reduction → Compression → Aggregation → Micro Macro Linkage
Neighborhood in Abstraction Space¶
Linear Discriminant Analysis sits in a sparse region of the domain-specific corpus (86th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.
Family — Unclustered & Miscellaneous (1565 abstractions)
Nearest neighbors
- Violin Plot — 0.83
- Boosting — 0.83
- Quantification (machine learning) — 0.80
- Polynomial Chaos Expansion — 0.79
- AdaBoost — 0.78
Computed from structural-signature embeddings · 2026-09-08