Linear least squares¶

Approximation of an overdetermined or rank-deficient linear system by choosing parameters that minimize a quadratic residual norm.

Version: v2 · 2026-09-19 · History
Domain-specific #: 5341
Domain group: Formal Sciences
Origin domain: Mathematics
Subdomain: Numerical Linear Algebra → Mathematics
Aliases: LLS

Core Idea¶

Given A and b, ordinary LLS minimizes the Euclidean norm of Ax minus b, while weighted and generalized forms change the residual metric; uniqueness requires appropriate rank or regularization conditions. Orthogonal projection maps the observation vector onto the column space of the design matrix, and QR, SVD or related factorizations compute a minimizer without requiring unstable normal-equation inversion. The abstraction is therefore identified by a declared carrier, a transformation or constraint over that carrier, and an invariant that tells an analyst whether the named structure is genuinely present.

How would you explain it like I'm…

Closest Stick Game

You have some dots on paper and want one straight stick that sits as close to all of them as it can. You can't touch every dot, so you measure how far off each one is. You pick the stick where those misses, made extra big when they're far, add up to the smallest total.

Best Fit by Squared Misses

Sometimes you have more measurements than unknowns, and no single answer fits every measurement exactly. Linear least squares picks the answer whose predictions miss the measurements by the smallest total, where each miss is squared before adding. Squaring makes big misses count a lot and keeps misses in either direction from cancelling out. Fitting a straight line through scattered dots is one common use. Sometimes more than one answer is equally good, and extra rules are needed to choose one.

Best Fit by Projection

In Linear least squares you have a matrix A (the design matrix) and a vector b (the observations), and you look for x that makes Ax as close as possible to b. 'Close' usually means the smallest sum of squared differences, which is the Euclidean length of the residual Ax − b. Geometrically, the best Ax is the orthogonal projection of b onto the set of all possible Ax, the column space of A. The answer is unique only if A's columns are independent (full column rank) or extra regularization is added. Weighted versions count some measurements more than others. Good computer methods use factorizations like QR or SVD rather than inverting AᵀA directly, because that inversion can be numerically unstable.

Ordinary Linear least squares solves min_x ‖Ax − b‖₂ for a design matrix A and response b. Weighted and generalized versions replace the Euclidean norm with a metric defined by a weight or covariance matrix. The minimizer makes Ax the orthogonal projection of b onto col(A), so the residual is orthogonal to the column space; this condition is equivalent to the normal equations AᵀAx = Aᵀb. A unique solution needs full column rank. Otherwise you must state a convention, such as the minimum-norm solution, or add regularization. Numerically, QR or SVD are preferred over forming and inverting AᵀA, because forming AᵀA squares the condition number. A complete specification states the dimensions, rank, residual orientation, norm or weights, any constraints, the uniqueness convention, the method and its conditioning. In statistical use it also states the error model and diagnostics.

Scope of Application¶

Linear least squares belongs to numerical linear algebra and is useful where the analyst can specify the typed numerical linear algebra carrier, including its objects, relations, parameters, conventions, evidence, boundary cases, and comparison targets, then evaluate the design matrix and response, dimensions and rank, residual orientation, norm or weight matrix, parameter constraints, minimizer and uniqueness convention, numerical method, conditioning, uncertainty model if statistical and diagnostics are explicit. The scope is broad within that domain but bounded by the need for the design matrix and response, dimensions and rank, residual orientation, norm or weight matrix, parameter constraints, minimizer and uniqueness convention, numerical method, conditioning, uncertainty model if statistical and diagnostics are explicit.

Clarity¶

The abstraction clarifies a crowded vocabulary by making the design matrix and response, dimensions and rank, residual orientation, norm or weight matrix, parameter constraints, minimizer and uniqueness convention, numerical method, conditioning, uncertainty model if statistical and diagnostics are explicit the center of the account. A claim should name the carrier, the governing operation or relation, the applicable assumptions, and the recognition test.

Manages Complexity¶

Without the abstraction, an analyst must reason directly over many local details: the carrier roles, admissibility assumptions, competing conventions, derived invariants, boundary cases, and proof or validation obligations specific to Linear least squares. Linear least squares compresses them into the roles in the structural signature. That compression permits comparison across instances without erasing the variables that determine validity. It also exposes which details may be varied safely and which are constitutive.

Abstract Reasoning¶

Identify the carrier. State what the elements, states, objects, or observations are: the typed numerical linear algebra carrier, including its objects, relations, parameters, conventions, evidence, boundary cases, and comparison targets. Reject examples whose alleged carrier belongs to a different problem.
Lock the constitutive rule. Express the design matrix and response, dimensions and rank, residual orientation, norm or weight matrix, parameter constraints, minimizer and uniqueness convention, numerical method, conditioning, uncertainty model if statistical and diagnostics are explicit independently of one notation or implementation.

Knowledge Transfer¶

Knowledge transfers strongly among subfields of numerical linear algebra because they reuse the typed numerical linear algebra carrier, including its objects, relations, parameters, conventions, evidence, boundary cases, and comparison targets, Orthogonal projection maps the observation vector onto the column space of the design matrix, and QR, SVD or related factorizations compute a minimizer without requiring unstable normal-equation inversion., and type the carrier, state every parameter and convention in the definition, test that the design matrix and response, dimensions and rank, residual orientation, norm or weight matrix, parameter constraints, minimizer and uniqueness convention, numerical method, conditioning, uncertainty model if statistical and diagnostics are explicit, compare the nearest accepted identity, and report counterexamples, uncertainty, and limiting cases.

Relationships to Other Abstractions¶

Current abstraction Linear least squares Domain-specific

Parents (1) — outgoing typed relations

Linear least squares is a kind of Approximation Prime

It remains its own entry because its identity is fixed by the design matrix and response, dimensions and rank, residual orientation, norm or weight matrix, parameter constraints, minimizer and uniqueness convention, numerical method, conditioning, uncertainty model if statistical and diagnostics.

Hierarchy path (1) — routes to 1 parentless root

Linear least squares → is a kind of Approximation → is a decomposition of Representation → presupposes Abstraction

Neighborhood in Abstraction Space¶

Linear least squares sits in a crowded region of the domain-specific corpus (11^th percentile for distinctiveness): several abstractions share nearly its structure, so a description that fits it tends to fit its neighbors too.

Family — Structured Matrices & Linear Maps (43 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-10-25