Gauss–Newton Algorithm¶
An iterative nonlinear least-squares algorithm that linearizes the residual vector and solves the resulting Jacobian least-squares subproblem for each parameter update.
Core Idea¶
The Gauss–Newton algorithm solves a smooth nonlinear least-squares problem by repeatedly replacing its nonlinear residual map with a first-order linear model. Given residuals \(r:\mathbb R^n\to\mathbb R^m\) and
let \(J_k\) be the \(m\times n\) Jacobian of \(r\) at the current iterate \(\beta_k\). The method chooses a step \(\Delta_k\) that solves
and updates \(\beta_{k+1}=\beta_k+\Delta_k\). If \(J_k\) has full column rank, the idealized step satisfies.
Scope of Application¶
Gauss–Newton is used for nonlinear regression, parameter estimation, curve fitting, inverse problems, bundle adjustment, system identification, geodesy, and overdetermined systems of nonlinear equations. It is especially attractive when first derivatives are accessible but residual Hessians are expensive or inconvenient.
Weighted least squares can be reduced to the same structure after applying a weighting or whitening operator to the residuals and Jacobian. Constraints, robust losses, bounds, and sparse structure require explicit extensions; their solvers may retain a Gauss–Newton model while no longer being the bare algorithm.
Clarity¶
Sign conventions must be stable. This dossier defines \(J=\partial r/\partial\beta\), giving \(r+J\Delta\) and \(J^{\mathsf T}J\Delta=-J^{\mathsf T}r\). If residuals are defined as observations minus predictions while \(J\) differentiates predictions, the local expression becomes \(r-J\Delta\) and the right-hand sign reverses.
Manages Complexity¶
Direct Newton optimization of \(S\) requires all second derivatives of all residuals. Gauss–Newton compresses that curvature calculation into the Gram matrix \(J^{\mathsf T}J\), which is assembled from first derivatives already needed to understand local sensitivity.
Each nonlinear iteration becomes a familiar linear least-squares problem. This permits mature dense, sparse, and structured linear algebra, and exposes parameter identifiability through singular values of the Jacobian.
Abstract Reasoning¶
Taylor expansion gives
Substituting the linear part into \(S\) yields the quadratic model
Differentiating gives \(\nabla m_k(\Delta)=J_k^{\mathsf T}(r_k+J_k\Delta)\), so stationarity produces the normal equations. When \(J_k\) has full column rank, \(J_k^{\mathsf T}J_k\) is positive definite and the modeled step is unique.
Knowledge Transfer¶
The mechanism transfers literally across models because a residual map supplies a common interface. An exponential decay fit, camera reprojection problem, chemical kinetic fit, and nonlinear calibration all form \(r\), compute \(J\), solve the local least-squares problem, and update parameters.
Automatic differentiation changes how \(J\) is obtained, not the algorithm's identity. Sparse QR changes the step implementation, not the local model. Damping and trust regions modify the identity enough to require variant labels, though they preserve a Gauss–Newton model internally.
Relationships to Other Abstractions¶
Current abstraction Gauss–Newton Algorithm Domain-specific
Parents (1) — more general patterns this builds on
-
Gauss–Newton Algorithm is a kind of Algorithm Prime
prime:algorithm is the proposed minimal parent by strict specialization.
Hierarchy paths (2) — routes to 2 parentless roots
- Gauss–Newton Algorithm → Algorithm → Function (Mapping)
Neighborhood in Abstraction Space¶
Gauss–Newton Algorithm sits in a sparse region of the domain-specific corpus (85th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.
Family — Numerical Discretization & Element Methods (6 abstractions)
Nearest neighbors
- Nonlinear Least Squares — 0.85
- Least absolute deviations — 0.80
- Exponential Integrator — 0.80
- Moreau Envelope — 0.79
- Condition Number — 0.79
Computed from structural-signature embeddings · 2026-09-08