Nonlinear Least Squares¶

Estimate parameters that enter a model nonlinearly by minimizing a residual sum of squares, usually through initialization-sensitive local iterations built from the residual Jacobian.

Version: v2 · 2026-09-06 · History
Domain-specific #: 2388
Origin domain: statistics
Subdomain: nonlinear regression
Aliases: Non Linear Least Squares

Core Idea¶

Nonlinear least squares estimates an unknown parameter vector \(\beta\) by minimizing a sum of squared residuals when the model is nonlinear in at least one unknown parameter. Given observations \((x_i,y_i)\) and a model response \(f(x_i;\beta)\), define

\[ r_i(\beta)=y_i-f(x_i;\beta), \qquad S(\beta)=\frac12\sum_{i=1}^{m}r_i(\beta)^2. \]

The factor \(1/2\) is optional and only simplifies derivatives. Weighted nonlinear least squares replaces the Euclidean residual norm by a declared positive weight or precision metric, equivalently by whitening induced by the error covariance. Correlated errors require the generalized or whitened counterpart rather than arbitrary diagonal weights. The defining commitments are nonlinearity in the unknowns and a squared-residual objective—not any particular solver.

The parameter distinction matters. The model \(\beta_0+\beta_1x+\beta_2x^2\) is linear in its unknown coefficients and is therefore linear least squares despite being curved in \(x\). The model \(\beta_1e^{-\beta_2x}\) is nonlinear in \(\beta_2\); its gradient equations normally have no closed-form solution. NIST uses this unknown-parameter criterion and treats smoothness and unique solution as practically helpful rather than definitional requirements.^[1]

Most solvers begin at \(\beta_k\), construct a local approximation from the Jacobian, calculate a step \(\Delta_k\), and update \(\beta_{k+1}=\beta_k+\Delta_k\). If \(J_k\) is the Jacobian of the model predictions, then

\[ f(\beta_k+\Delta)\approx f(\beta_k)+J_k\Delta, \]

so a Gauss–Newton step approximately solves

\[ \min_\Delta \lVert r_k-J_k\Delta\rVert_2^2. \]

Levenberg–Marquardt and trust-region variants damp or constrain this local step when the quadratic approximation is unreliable. Rust's NIST treatment presents Gauss–Newton and Levenberg–Marquardt together with fit diagnostics in scientific curve fitting.^[2]

The result is local and conditional. Starting values can lead to different minima, poor scaling can destabilize steps, a rank-deficient Jacobian at a fitted point can diagnose weak local sensitivity or practical identification, and squared residuals make the fit sensitive to outliers. Structural nonidentifiability requires a model-wide analytic or design argument. Solver termination means a stopping criterion was met; it does not by itself prove global optimality or scientific adequacy.

The abstraction is domain-specific. Residual vectors, parameter Jacobians, squared-error objectives, local curvature, statistical error models, and approximate covariance transfer literally across scientific fitting and numerical estimation, but not across arbitrary nonnumeric optimization. Optimization carries the portable genus.

Structural Signature¶

Sig role-phrases:

the observations and design points — measured responses and their experimental, spatial, temporal, or covariate settings
the parameterized nonlinear model — a response or residual map with at least one unknown entering nonlinearly
the parameter vector and admissible region — unknown values to estimate, including any bounds, transformations, or physical restrictions
the residual vector — observed minus predicted values, defined consistently with any weights or covariance transformation
the squared-residual objective — the scalar criterion \(S(\beta)=\frac12\lVert r(\beta)\rVert^2\) whose minimizer defines the estimate
the Jacobian or local model — first-order sensitivity of predictions or residuals to parameter changes
the step and globalization rule — Gauss–Newton, damping, trust region, line search, or another method that controls movement from the current iterate
the initialization and convergence basin — the starting state and local geometry that govern which stationary point can be reached
the stopping and adequacy diagnostics — gradient, step, objective, rank, scaling, residual, and sensitivity checks separating numerical termination from a defensible fit

Sign conventions must be explicit. This entry defines \(r=y-f(\beta)\) and uses \(J=\partial f/\partial\beta\), giving the local residual \(r-J\Delta\) and normal-equation form \(J^{\mathsf T}J\Delta=J^{\mathsf T}r\). If \(J\) instead denotes \(\partial r/\partial\beta\), the sign changes. Either convention is correct when used consistently.

The identity allows several solver families. Repeated local linear least-squares updates are characteristic, but an NLS problem remains the same optimization problem if solved by a trust-region, variable-projection, global-search, or derivative-free procedure. Solver behavior belongs to the implementation contract; squared residuals and nonlinear parameter dependence define the problem.

What It Is Not¶

Not any nonlinear-looking curve. Polynomial regression is nonlinear in \(x\) but linear in its coefficients. Tell: do the unknown parameters enter through a linear combination, or inside exponentials, denominators, powers, phases, or other nonlinear operations?
Not generic nonlinear regression. Nonlinear regression can use likelihood, robust, quantile, or other loss functions. NLS specifically minimizes squared residuals.
Not one solver. Gauss–Newton and Levenberg–Marquardt are procedures for NLS; neither is the full problem class.
Not residual analysis. Residual analysis studies leftover structure and model adequacy after or around fitting. NLS uses residual magnitudes to define the estimator even if no diagnostic plot is made.
Not guaranteed global optimization. Standard local methods can converge to different stationary points from different starts or stop in poorly scaled, nearly flat regions.
Not ordinary linear least squares with more iterations. Linear least squares has a globally quadratic objective and can be solved directly by QR or SVD. NLS repeatedly changes its local model because the Jacobian depends on the parameters.
Not automatically maximum likelihood. With independent Gaussian errors of common known structure, minimizing the corresponding weighted sum of squares agrees with maximum likelihood. Other error distributions do not generally license that interpretation.
Not automatically identified. A low objective value can coexist with a ridge of parameter combinations that make nearly identical predictions. Numerical convergence does not prove parameter uniqueness.
Not robust to outliers. A quadratic-loss contribution grows with squared residual magnitude, while its score or influence is unbounded and grows linearly with residual magnitude. Replacing the quadratic loss with Huber, absolute, or another robust loss creates a related but different estimator.

Scope of Application¶

Scientific curve fitting. Exponential decay, saturation, growth, dose-response, binding, kinetic, compartmental, and periodic models often encode mechanistic parameters directly. Their nonlinear parameter placement preserves physical meaning that a convenient polynomial approximation may lose.

Calibration and metrology. Instrument response curves, sensor transfer functions, spectral peaks, and reference-material models are fitted to standards. Parameter correlation, design coverage, weights, and traceable uncertainty are as important as the residual minimum.

Engineering identification. Dynamic-system, frequency-response, thermal, material, and reliability models map physical parameters into predictions. Bounds and transformations can enforce positivity or other feasible physics.

Pharmacology and biochemistry. Michaelis–Menten, Hill, Emax, and multi-exponential models use nonlinear parameters for rates, capacities, affinities, and slopes. Sparse experimental design can make combinations of these parameters weakly identifiable.

Econometrics and statistical regression. Threshold, smooth-transition, nonlinear production, and transformed-regressor models may be estimated by NLS when conditional-mean and error assumptions justify squared loss. Binary-response likelihood models are nonlinear regression but are not ordinarily NLS simply because their link is nonlinear.

Geodesy and photogrammetry. Observation equations are often nonlinear in coordinates, orientations, or calibration parameters and are linearized within a weighted least-squares adjustment. The broader adjustment package adds condition equations, datum, covariance propagation, and network diagnostics.^[3]

Computer vision. Bundle adjustment and geometric calibration minimize nonlinear reprojection errors. Sparse Jacobian structure, gauge freedoms, robust losses, and large-scale trust-region methods are load-bearing; robustified bundle adjustment is not pure quadratic NLS even when it begins from this template.

The scope ends when there is no residual-square objective or no nonlinear parameter dependence. A neural-network cross-entropy fit, a linear polynomial regression, and post-fit residual inspection belong elsewhere despite sharing pieces of the computation.

Clarity¶

The abstraction exposes five commitments hidden by “we fit a nonlinear curve”: what was measured, which functional form was selected, which parameters enter nonlinearly, what residual metric was minimized, and how the local solver was initialized and judged. Two analyses can use the same data and curve family yet obtain different results because parameter bounds, weights, starts, or stopping rules differ.

It also separates three kinds of success. Numerical success means an algorithm met tolerances. Optimization success means a sufficiently good—ideally global—minimum of the declared objective was found. Modeling success means the model and error assumptions support the intended prediction or interpretation. A small step can indicate convergence or merely bad scaling; a small residual sum can indicate a good model or overfitting; a stable prediction can coexist with unstable individual parameters.

The unknown-parameter test prevents a frequent classification error. A high-degree polynomial may look nonlinear on a plot but its coefficients can be estimated by one linear least-squares solve. An exponential with an unknown decay rate requires NLS because changes in that rate alter the Jacobian and curvature. This distinction predicts computation, initialization sensitivity, and uncertainty geometry before any code runs.

Finally, the local-versus-global distinction disciplines reporting. “The solver converged” should be expanded into starting values, algorithm, tolerances, gradient or projected-gradient norm, objective value, Jacobian rank/condition, and multi-start or alternative-solver evidence when multiple minima are plausible.

Manages Complexity¶

Directly solving the nonlinear first-order conditions can be intractable. NLS manages that difficulty by preserving the global residual objective while repeatedly substituting a local linear least-squares subproblem. Mature QR, SVD, and linear-system tools can then produce a candidate step. The model is reconsidered at the new parameter value, so curvature is handled by successive approximation rather than one symbolic solution.

Gauss–Newton exploits special structure. With \(r_i=y_i-f_i(\beta)\) and prediction Jacobian \(J=\partial f/\partial\beta\), the exact Hessian is

\[ \nabla^2S(\beta)=J^{\mathsf T}J-\sum_i r_i(\beta)\,\nabla^2 f_i(\beta). \]

Ignoring the residual-weighted prediction Hessians is effective near a good fit with modest residuals, but can be unreliable far from the solution or under strong curvature. Levenberg–Marquardt adds damping, schematically \((J^{\mathsf T}J+\lambda D)\Delta=J^{\mathsf T}r\), where \(\lambda\ge0\) and \(D\) is an appropriate positive damping or scaling matrix, to shorten or redirect unstable steps; Moré develops its implementation and convergence logic.^[4]

The structural signature localizes failure. If steps explode, inspect scaling, bounds, and damping. If the objective barely changes along a direction, inspect Jacobian rank and parameter correlation. If different starts reach different fits, map basins or use multi-start/global strategies. If a few points dominate, inspect leverage and change the error model or robust loss rather than only tuning the solver.

Computation should avoid worsening conditioning without need. Explicitly forming \(J^{\mathsf T}J\) squares the condition number in the normal-equation representation; QR, SVD, or structured factorizations can be safer. The normal equations explain the step but are not a universal implementation recommendation.

Abstract Reasoning¶

Parameter-linearity prediction. If all unknowns enter linearly, one fixed design matrix suffices and local relinearization is unnecessary. Reclassify the problem as linear least squares even if the predictor transformation is nonlinear.

Starting-value prediction. If the objective has multiple basins or a narrow curved valley, different plausible starts can produce different parameter estimates. A single run gives weak evidence; multi-start agreement or domain-informed initialization increases confidence.

Small-residual prediction. When residuals are small and the Jacobian has full rank near the solution, Gauss–Newton's omitted second-derivative terms become less influential and fast local convergence is plausible. Large residuals warn that this approximation may be poor.

Rank prediction. If columns of the Jacobian at the fitted point are linearly dependent or nearly so, some parameter combinations have weak local sensitivity or are practically unidentified there. This local diagnostic does not alone prove structural nonidentifiability; that stronger conclusion requires a model-wide analytic or design argument. More iterations cannot create information absent from the design; reparameterization or new observations may be required.

Scaling prediction. Parameters with very different numerical scales can make Euclidean steps and stopping rules misleading. Scaling, transformations, or a diagonal damping matrix can change computational behavior without changing the underlying scientific model.

Outlier prediction. Each quadratic-loss contribution grows with squared residual magnitude, while its score or influence is unbounded and grows linearly with residual magnitude; a single extreme point can therefore move the optimum sharply. Rechecking the observation, specifying heteroscedastic weights, or using a robust loss is an inferential intervention, not merely an optimizer setting.

Uncertainty prediction. A covariance approximation such as \(\hat\sigma^2(J^{\mathsf T}J)^{-1}\) requires local full rank and an appropriate independent homoscedastic error model, or its weighted/generalized counterpart. Strong curvature, boundaries, weak identification, or multiple minima make symmetric local intervals unreliable.^[1]

Knowledge Transfer¶

The mechanism transfers intact across kinetics, growth curves, geodesy, calibration, spectral analysis, econometrics, and computer vision. Surface equations change, but the roles remain: observations, nonlinear parameter map, residual objective, Jacobian, starting point, local step, stopping rule, and fit/identification diagnostics.

Knowledge about initialization and scaling travels especially well. A decay model with a poor rate start and a camera-calibration problem with a poor pose start fail for the same structural reason: the current local model points into an unhelpful basin. Parameter transformations, staged fitting, coarse grid starts, and multi-start checks port between those fields without metaphor.

The transfer hazard is carrying statistical interpretations without their assumptions. A numerical residual minimum exists independently of an error distribution. Standard errors, likelihood language, confidence intervals, and efficiency claims require a stochastic model. Conversely, a statistically appropriate objective can still be solved poorly. Keep numerical and inferential warrants separate.

Outside numeric parameter fitting, “iterate toward a better fit” is only the broader Optimization/Iteration skeleton. Product design and policy refinement do not instantiate Nonlinear Least Squares unless they declare a nonlinear parameterized residual map and minimize its squared norm.

Examples¶

Canonical¶

Fit \(f(x;a,b)=ae^{-bx}\) to two observations \((0,2)\) and \((1,1)\). Start at \((a_0,b_0)=(1,0)\). Predictions are \((1,1)\), so \(r=(1,0)\) and the unhalved residual sum of squares is \(1\). The prediction Jacobian is

\[ J=\begin{bmatrix} e^{-bx} & -axe^{-bx} \end{bmatrix}_{(a,b)=(1,0)} =\begin{bmatrix}1&0\\1&-1\end{bmatrix}. \]

The local Gauss–Newton problem is \(\min_\Delta\lVert r-J\Delta\rVert^2\). Here the exact local solution is \(\Delta=(1,1)\), giving \((a_1,b_1)=(2,1)\). New predictions are \((2,2/e)\), and the residual sum falls to \((1-2/e)^2\approx0.0698\). The exact two-point fit is \(a=2,b=\log2\); later iterations can move toward it. The example demonstrates improvement, not a general one-step guarantee.

Mapped back: the two pairs are the observations/design; \(ae^{-bx}\) is the nonlinear model; \((a,b)\) is the parameter vector; \((1,0)\) is the residual; its squared norm is the objective; the displayed matrix is the Jacobian; \((1,1)\) is the local step; the starting pair supplies initialization; and the objective drop is one progress diagnostic rather than proof of global convergence.

Applied / In Practice¶

A materials laboratory measures concrete strength at several curing times and fits an asymptotic curve \(f(t;A,k)=A(1-e^{-kt})\). NIST uses concrete-strength behavior as an example of a process whose quick rise and leveling are poorly represented by a straight line.^[1] \(A\) controls the long-run strength and \(k\) the approach rate. When observations cover only early times, increasing \(A\) while decreasing \(k\) can preserve nearly the same predicted curve, producing a narrow Jacobian valley and unstable separate estimates. Adding later-time measurements targets the missing asymptote information.

The analyst uses positive parameter transformations, several starts, a damped Gauss–Newton/Levenberg–Marquardt solver, and residual plots. Agreement in predicted strength across starts with disagreement in \(A\) is reported as practical nonidentifiability rather than hidden by one converged run. If a cracked specimen is an outlier, the team investigates it and performs a robust-loss sensitivity fit instead of assuming the quadratic objective is immune.

Mapped back: curing-time measurements are the observations/design; the saturation equation is the nonlinear model; \(A,k\) and positivity bounds are the parameters/region; observed-minus-predicted strengths are the residuals; their squares define the objective; sensitivities to \(A,k\) form the Jacobian; damping controls the step; multi-start probes the basin; and rank, residual, profile, and held-out checks provide stopping and adequacy diagnostics.

Structural Tensions¶

T1: Mechanistic meaning versus numerical identifiability. A nonlinear equation can give parameters rich scientific interpretations while the available design identifies only combinations of them. Diagnostic: do Jacobian singular values and profile objectives support each named parameter separately?

T2: Fast Gauss–Newton progress versus curvature safety. Aggressive local steps can converge quickly near a good solution and fail far away. Diagnostic: does the actual objective reduction agree with the local model's predicted reduction, or is damping/trust control needed?

T3: One good start versus basin uncertainty. A single plausible initialization is cheap but can conceal alternative minima. Diagnostic: do dispersed, domain-feasible starts converge to the same objective and scientifically equivalent parameters?

T4: Squared-error efficiency versus outlier fragility. Quadratic loss is convenient and efficient under Gaussian-like errors but gives extreme observations large influence. Diagnostic: do leave-one-out, influence, and robust-loss sensitivity checks change the conclusion?

T5: Parameter bounds versus inferential geometry. Bounds enforce physical plausibility yet can place estimates on boundaries where ordinary covariance approximations fail. Diagnostic: is a parameter interior with locally symmetric uncertainty, or boundary-limited and in need of profiles/bootstrap methods?

T6: Numerical convergence versus model adequacy. A tiny gradient certifies only a local property of the chosen objective. Diagnostic: are residual structure, validation, and scientific plausibility acceptable independently of solver tolerances?

T7: Flexible curve family versus extrapolation stability. Nonlinear mechanisms can fit observed shapes well while diverging sharply outside the design range. Diagnostic: are prediction targets supported by observations that constrain the relevant asymptote, threshold, or phase?

T8: Autonomy versus reduction. NLS is Optimization using Iteration and residuals, but it owns the nonlinear-parameter/squared-residual/Jacobian/basin closure. Diagnostic: if that closure is absent, route to generic Optimization, Regression, or Iteration; if it remains load-bearing, preserve the domain-specific node.

Structural–Framed Character¶

Nonlinear Least Squares is mixed-structural. Its equations and local geometry are evaluatively neutral: a residual norm, Jacobian, stationary point, and condition number make no normative judgment. Once the model and data are fixed, the objective and derivatives are formal.

It is human-practice-bound as an estimation instrument. Analysts choose the model family, observations, weights, parameter transformations, starts, solver, tolerances, and intended interpretation. The optimization problem does not appear as a self-executing natural mechanism; it is an engineered way to extract parameters from measurements.

Its institutional origin lies in numerical analysis, statistics, and scientific measurement. Its vocabulary travels literally across technical fields, but those uses are repeated deployment of the same mathematical estimator rather than independent recurrence in arbitrary substrates. Within that technical band, transfer is recognition: a kineticist and a camera-calibration engineer face the same Jacobian rank, basin, scaling, and damping issues.

Outside numerical parameter estimation, only Optimization and Iteration travel. Calling organizational refinement “nonlinear least squares” would import mathematics without the defining residual map. Its character: a formally precise residual-minimization structure whose model choice, initialization, diagnostics, and inferential reading remain framed by scientific estimation practice.

Structural Core vs. Domain Accent¶

What is skeletal. Choose variables, define an objective, improve a candidate, carry state between updates, and diagnose progress. Those roles belong to prime:optimization and prime:iteration. Studying what the current model leaves behind belongs to Residual Analysis.

What is domain-bound. The variables are numerical model parameters; the objective is the squared norm of an observation residual vector; the local map is a parameter Jacobian; Gauss–Newton exploits \(J^{\mathsf T}J\); damping manages curvature; and covariance/identifiability claims depend on statistical error and design structure.

Why it does not clear the prime bar. Free substitution of policies, designs, biological states, or organizational choices removes the residual Jacobian, squared-error estimator, and parameter-covariance interpretation. The portable improvement loop is already represented by Optimization and Iteration. The distinctive mechanism remains confined to numerical/statistical parameter fitting.

Strictly specializes prime:optimization. NLS is minimization specialized to a residual-sum-of-squares objective generated by a model nonlinear in its parameters. Optimization supplies decision variables, objective, feasible region, and local/global distinction; NLS adds observations, residual map, Jacobian structure, and estimation diagnostics. This is the proposed sole direct subsumption/strict edge.

Commonly uses prime:iteration. Standard solvers carry \(\beta_k\) into the next local model and update, but iteration is an implementation pattern rather than the mathematical identity of every problem instance. Preserve as a prose relation unless audit finds a universal direct edge necessary.

Related to domain_specific:regression. When the residuals arise from an outcome model with a stochastic error contract, NLS is a regression fitting method. Generic numerical residual-map problems do not necessarily satisfy the live Regression node's full outcome/noise/use package, so no direct parent is proposed.

Distinct from prime:residual_analysis. NLS minimizes residual magnitudes to estimate parameters; Residual Analysis studies leftover structure to diagnose what a model missed. A workflow may use both, but neither relation is taxonomic.

Relationships to Other Abstractions¶

Current abstraction Nonlinear Least Squares Domain-specific

Parents (1) — more general patterns this builds on

Nonlinear Least Squares is a kind of Optimization Prime

Strictly specializes prime:optimization. NLS is minimization specialized to a residual-sum-of-squares objective generated by a model nonlinear in its parameters.

Hierarchy path (1) — routes to 1 parentless root

Nonlinear Least Squares → Optimization

Neighborhood in Abstraction Space¶

Nonlinear Least Squares sits in a sparse region of the domain-specific corpus (67^th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Statistical Adjustment & Estimation Effects (14 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-09-08

Not to Be Confused With¶

Linear least squares. The unknowns enter linearly and one fixed design matrix defines a globally quadratic problem. Tell: does the Jacobian change with the parameter estimate?
Nonlinear regression. This broader family can use nonquadratic likelihoods or losses. Tell: is a residual sum of squares specifically the defining estimator?
Gauss–Newton algorithm. A local solver exploiting the NLS Jacobian. Tell: is the residual-square problem being named, or one procedure used to solve it?
Levenberg–Marquardt algorithm. A damped local method balancing Gauss–Newton-like and shorter steps. Tell: is damping behavior constitutive, or could another NLS solver be substituted?
Residual Analysis. Examines the structure of leftovers and noise adequacy. Tell: are residuals being minimized to fit parameters or inspected for remaining pattern?
Maximum likelihood estimation. Optimizes a probability model's likelihood. Tell: has a Gaussian error contract made likelihood proportional to squared loss, or is that bridge absent?
Orthogonal distance regression / total least squares. Allows error in explanatory coordinates and measures a different geometric residual. Tell: are only response residuals squared, or are adjustments made in multiple coordinate directions?
Least-squares adjustment. Reconciles redundant observations with parametric, conditional, or combined equations, weights, datum, and covariance propagation. Tell: is the object a general nonlinear residual fit, or the fuller observation-adjustment workflow?
Robust nonlinear fitting. Replaces quadratic loss with a bounded-influence or heavy-tailed criterion. Tell: do large residuals retain quadratic influence or receive robust downweighting?

References¶

[1] NIST/SEMATECH, “Nonlinear Least Squares Regression,” e-Handbook of Statistical Methods, §4.1.4.2. https://www.itl.nist.gov/div898/handbook/pmd/section1/pmd142.htm. Verified 2026-08-26. registry ↩a ↩b ↩c

[2] Bert W. Rust, “Fitting Nature's Basic Functions Part III: Exponentials, Sinusoids and Nonlinear Least Squares,” Computing in Science & Engineering 4(4), 2002. https://www.nist.gov/publications/fitting-natures-basic-functions-part-iii-exponentials-sinusoids-and-nonlinear-least. Verified 2026-08-26. registry ↩

[3] Allan Aasbjerg Nielsen, Least Squares Adjustment: Linear and Nonlinear Weighted Regression Analysis, Technical University of Denmark, 2013. https://www2.imm.dtu.dk/pubdb/edoc/imm2804.pdf. Verified 2026-08-26. registry ↩

[4] Jorge J. Moré, “The Levenberg–Marquardt Algorithm: Implementation and Theory,” in Numerical Analysis, Lecture Notes in Mathematics 630, 1978. https://doi.org/10.1007/BFb0067700. Verified 2026-08-26. registry ↩