Automatic Differentiation¶

A family of program-evaluation and transformation techniques that decomposes an executed numerical computation into differentiable primitives and composes their local derivative rules to obtain derivatives accurate to working precision, chiefly through forward Jacobian–vector or reverse vector–Jacobian accumulation.

Version: v2 · 2026-09-19 · History
Domain-specific #: 1322
Domain group: Formal Sciences
Origin domain: Mathematics
Subdomains: Numerical Computing, Derivative Computation → Mathematics
Aliases: Algorithmic differentiation, Autodiff, Auto-differentiation, AD

Core Idea¶

Automatic differentiation (AD) is a family of techniques for evaluating derivatives of numerical functions represented by computer programs. It observes that an executed computation is a composition of elementary operations—addition, multiplication, division, trigonometric functions, exponentials, linear-algebra primitives, and other operations supplied with derivative rules. AD applies the chain rule to those operations in the order required by the program, producing derivatives of the represented computation without first converting the whole program into a symbolic formula and without perturbing inputs by a finite step.

How would you explain it like I'm…

The Step-by-Step Change Finder

A computer program does math in lots of tiny steps, like adding and multiplying. Automatic differentiation is a trick where, for each tiny step, the computer already knows the rule for how much the answer would change if the start changed, and it links those rules together in a chain. So it finds out exactly how sensitive the answer is without guessing by trying.

Chain-Rule Calculator for Programs

A computer program that does math is really a long chain of small steps, like adding, multiplying, or taking a square root. For every small step, there's a simple rule for how much its output changes when its input changes. Automatic differentiation has the computer apply those rules step by step and link them together, which tells it how the final answer depends on the starting numbers. It doesn't guess by trying slightly different numbers, and it doesn't first write out one giant formula. The result is as accurate as the computer's own arithmetic allows, but it only works if every step has a correct rule.

Program-Level Derivative Propagation

Automatic differentiation (AD) computes derivatives of functions written as computer programs. Any executed calculation is a composition of elementary operations — addition, multiplication, exponentials, sine, matrix operations — each with a known local derivative rule. AD applies the chain rule to these operations in program order. That makes it different from symbolic differentiation (which builds a whole formula first) and from finite differences (which nudge inputs by a small step and compare, introducing approximation error). AD's answer is exact up to ordinary floating-point rounding, but 'automatic' doesn't mean it can differentiate anything: it can still be tripped up by non-differentiable points, unstable calculations, or a wrong custom rule. Forward mode carries derivatives along with the calculation and suits few inputs; reverse mode runs backward from the output and suits many inputs with one output, like the gradient used to train a neural network.

Automatic differentiation (AD) evaluates derivatives of numerical functions represented as programs by exploiting the fact that an executed computation is a composition of elementary operations (arithmetic, transcendental functions, linear-algebra primitives, and others supplied with derivative rules). It applies the chain rule to these operations in program order, avoiding both symbolic expansion of the whole program and finite-difference perturbation. 'Automatic' means mechanical propagation, not license to differentiate anything: a valid AD system needs a program trace or intermediate representation, differentiable primitives with correct local rules, designated inputs, and a mode for composing rules. The output is the derivative of the executed computation at the given point, exact to working precision: it avoids truncation and cancellation error from finite differences, but not roundoff, nondifferentiability, unstable primal computations, or incorrect custom rules. For f: Rⁿ → Rᵐ with Jacobian J_f(x), forward mode computes a Jacobian–vector product J_f(x)v alongside f(x), while reverse mode records or reconstructs the primal computation and propagates an output cotangent backward to compute a vector–Jacobian product uᵀJ_f(x). Forward mode is efficient when few input directions are needed; reverse mode when few output directions are needed, notably gradients of scalar objectives with many parameters.

Scope of Application¶

AD recurs across scientific computing, nonlinear optimization, optimal control, computational physics, inverse problems, statistics, computer graphics, robotics, finance, and machine learning. These practices share a need for gradients, Jacobians, Hessian products, and parameter sensitivities of calculations too large or changeable for manual derivatives.

In machine learning, reverse mode differentiates a scalar loss with respect to many parameters. In simulation and optimization, forward mode can propagate selected parameter or state directions, while reverse/adjoint mode can obtain sensitivities of a small number of objectives to many inputs.

Clarity¶

A claim of AD should specify five things:

Primal function and active inputs. Which arguments are differentiated, at which values, and which are treated as constants?
Mode and seed. Is the result a JVP \(Jv\), VJP \(u^{\mathsf T}J\), scalar gradient, dense Jacobian, or nested higher derivative?
Trace semantics. Is the graph static, traced dynamically, transformed from source, or built by operator overloading?

Manages Complexity¶

The key compression is reuse of the primal program's decomposition. Instead of deriving and maintaining one monolithic formula, an AD system maintains local rules for a comparatively small primitive set. Every program assembled from those primitives inherits derivative evaluation through the chain rule. When the primal code changes, derivative behavior changes with the trace, reducing manual synchronization errors.

Abstract Reasoning¶

For an operation \(w_i=\phi_i(w_{p_1},\ldots,w_{p_k})\), forward mode propagates a tangent

\[ \dot w_i=\sum_{j=1}^{k}\frac{\partial\phi_i}{\partial w_{p_j}}\dot w_{p_j}. \]

Starting with input seed \(\dot x=v\), the final tangent is \(\dot y=J_f(x)v\). One seed gives one directional derivative or one Jacobian column when \(v\) is a basis vector.

Knowledge Transfer¶

Literal transfer occurs whenever a numerical program is composed from supported differentiable primitives. The same JVP/VJP machinery applies to a neural network, an ODE solver, a ray tracer, a portfolio valuation, or a physics simulation. What changes is the primitive library, trace, active parameters, and domain validation; chain-rule accumulation remains literal.

Implementation strategies transfer too. Operator overloading is convenient for dynamic languages, source transformation can enable ahead-of-time optimization, and compiler intermediate representations can make control/data dependencies explicit.

Relationships to Other Abstractions¶

Current abstraction Automatic Differentiation Domain-specific

Parents (1) — outgoing typed relations

Automatic Differentiation presupposes Derivative Domain-specific

Automatic Differentiation presupposes Derivative.

Hierarchy paths (2) — routes to 2 parentless roots

Automatic Differentiation → presupposes Derivative → presupposes Function (Mapping)

Show alternative path (1)

Neighborhood in Abstraction Space¶

Automatic Differentiation sits in a sparse region of the domain-specific corpus (93^rd percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Unclustered & Miscellaneous (2769 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-10-25