Automatic Differentiation¶

A family of program-evaluation and transformation techniques that decomposes an executed numerical computation into differentiable primitives and composes their local derivative rules to obtain derivatives accurate to working precision, chiefly through forward Jacobian–vector or reverse vector–Jacobian accumulation.

Version: v1 · 2026-08-30 · History
Domain-specific #: 1322
Origin domain: numerical computing
Subdomain: derivative computation
Aliases: Algorithmic differentiation, Autodiff, Auto-differentiation, AD

Core Idea¶

Automatic differentiation (AD) is a family of techniques for evaluating derivatives of numerical functions represented by computer programs. It observes that an executed computation is a composition of elementary operations—addition, multiplication, division, trigonometric functions, exponentials, linear-algebra primitives, and other operations supplied with derivative rules. AD applies the chain rule to those operations in the order required by the program, producing derivatives of the represented computation without first converting the whole program into a symbolic formula and without perturbing inputs by a finite step.^[1]^[2]

The adjective automatic names mechanical derivative propagation, not mathematical permission to differentiate anything. A valid AD system needs a program trace or intermediate representation, differentiable primitives with correct local rules, inputs with respect to which derivatives are requested, and a mode that composes those rules. The result is the derivative of the executed differentiable computation at the supplied point, subject to ordinary floating-point rounding and implementation semantics. It is therefore often described as exact to working precision: it avoids finite-difference truncation and cancellation, but it does not escape roundoff, nondifferentiability, unstable primal computation, or an incorrect custom rule.^[1]

For $f:\mathbb{R}^n\rightarrow\mathbb{R}^m$ with Jacobian $J_f(x)$, forward mode evaluates a Jacobian–vector product $J_f(x)v$ while evaluating $f(x)$. Reverse mode records or reconstructs the primal computation, then propagates an output cotangent backward to evaluate a vector–Jacobian product $u^{\mathsf T}J_f(x)$. Forward mode is naturally efficient when few input directions are needed; reverse mode is naturally efficient when few output directions are needed, especially the gradient of a scalar objective with many parameters.^[3]

Structural Signature¶

The defining flow is:

numerical program + input point + differentiable primitive rules + derivative seed -> executed primal trace -> chain-rule accumulation in forward or reverse mode -> JVP, VJP, gradient, Jacobian, or higher derivative + numerical/error qualifications

Eight roles are mandatory:

Primal program. A program or computational graph maps numerical inputs to outputs through an executed sequence of operations.
Differentiation point. Concrete primal values determine both local derivative values and, for dynamic control flow, the executed trace.
Primitive rule set. Each differentiable primitive has a trusted pushforward/JVP or pullback/VJP rule.
Dependency structure. Intermediate values record how outputs depend on inputs; reverse mode also needs saved values or a way to recompute them.
Seed direction or cotangent. Forward mode starts with input tangents; reverse mode starts with output adjoints.
Chain-rule accumulation. Local linear maps are composed along dependencies rather than a global expression being symbolically expanded.
Derivative product. The computation returns a directional derivative, gradient, Jacobian product, full Jacobian assembled from products, or a higher-order derivative built by nesting modes.
Semantic qualification. Control flow, mutation, random operations, discrete indexing, nondifferentiable primitives, custom rules, numeric type, and rounding determine what derivative was actually computed.

The invariant is: the derivative result is obtained by composing local derivative maps of the executed program, with the same dependency structure as its primal calculation. Merely returning a derivative value does not establish AD; analytic formulas, finite differences, complex-step approximation, or black-box surrogate fitting can return similar numbers by different mechanisms.

What It Is Not¶

Not the derivative itself. A derivative is the best local linear response of a function. AD is one computational method for evaluating it.
Not symbolic differentiation. Symbolic systems transform mathematical expressions and may simplify or expand a derivative expression. AD propagates numerical primal and derivative values through program operations, though source-transformation implementations can generate derivative code.
Not finite differences. Finite differences evaluate $f(x+h)-f(x)$ and balance truncation against cancellation. AD uses local derivative rules and no perturbation step $h$.
Not automatically exact real arithmetic. Floating-point primal values and derivative accumulations round; overflow, underflow, conditioning, and unstable algorithms still matter.
Not numerical differentiation in the perturbation sense. AD normally computes numerical derivative values, but its error mechanism differs from finite differencing.
Not synonymous with reverse mode. Forward, reverse, and mixed/nested modes all belong to AD.
Not synonymous with backpropagation. Neural-network backpropagation is a prominent reverse-mode application with a loss and parameter graph; reverse AD is the more general program-level mechanism.^[1]
Not proof of differentiability. At a branch boundary, absolute-value cusp, discrete index, comparison, or other nonsmooth operation, an implementation may choose a branch derivative, a convention, zero, a subgradient-like rule, or an error. The output does not make the mathematical function differentiable.
Not differentiation of arbitrary side effects. I/O, mutation, stateful randomness, and external calls need explicit semantics or transformation support.

Scope of Application¶

AD recurs across scientific computing, nonlinear optimization, optimal control, computational physics, inverse problems, statistics, computer graphics, robotics, finance, and machine learning. These practices share a need for gradients, Jacobians, Hessian products, and parameter sensitivities of calculations too large or changeable for manual derivatives.^[2]^[1]

In machine learning, reverse mode differentiates a scalar loss with respect to many parameters. In simulation and optimization, forward mode can propagate selected parameter or state directions, while reverse/adjoint mode can obtain sensitivities of a small number of objectives to many inputs. JAX exposes both JVP and VJP transformations and composes them for Hessian–vector products; PyTorch's autograd supplies reverse-mode graph differentiation and a developing forward-mode interface.^[3]^[4]

The scope is computational, not tied to one programming language or graph representation. Operator overloading can build a dynamic trace; source transformation or compiler passes can transform a program; tape-based systems can record intermediates. All qualify if derivative propagation follows the program's local differentiable operations. An algebraic derivation performed once by a human and hard-coded later does not become AD merely because software executes it.

Clarity¶

A claim of AD should specify five things:

Primal function and active inputs. Which arguments are differentiated, at which values, and which are treated as constants?
Mode and seed. Is the result a JVP $Jv$, VJP $u^{\mathsf T}J$, scalar gradient, dense Jacobian, or nested higher derivative?
Trace semantics. Is the graph static, traced dynamically, transformed from source, or built by operator overloading? How are loops, branches, mutation, and randomness handled?
Primitive coverage. Which operations have built-in or custom derivative rules, and what happens at unsupported or nondifferentiable operations?
Numerical and validation policy. What dtype, tolerances, conditioning, checkpointing, and independent derivative checks apply?

This diagnostic prevents common category errors. Saying “AD gives an exact derivative” should mean exact chain-rule propagation for the implemented primitive composition up to working arithmetic, not symbolic exactness. Saying “reverse mode computes the Jacobian” omits that one VJP produces a row combination; a full Jacobian generally requires multiple seeds. Saying “the gradient is zero” may describe a saturating or nondifferentiable implementation rule rather than the intended mathematical model.

Manages Complexity¶

The key compression is reuse of the primal program's decomposition. Instead of deriving and maintaining one monolithic formula, an AD system maintains local rules for a comparatively small primitive set. Every program assembled from those primitives inherits derivative evaluation through the chain rule. When the primal code changes, derivative behavior changes with the trace, reducing manual synchronization errors.^[2]

Mode choice avoids materializing large Jacobians. Forward mode carries one or several tangent directions through the computation, making its cost scale with the number of required input directions. Reverse mode propagates one or several output cotangents backward, making a scalar-output gradient available with a small-multiple arithmetic cost relative to the primal evaluation, but requiring access to intermediate primal data.^[1]^[3]

Reverse mode therefore creates a memory–recomputation tension. Retaining every intermediate simplifies the backward pass but can exhaust memory; checkpointing discards selected values and recomputes them later. This is not merely an implementation optimization: it is a structural consequence of applying transposed local derivatives in reverse dependency order.

Abstract Reasoning¶

For an operation $w_i=\phi_i(w_{p_1},\ldots,w_{p_k})$, forward mode propagates a tangent

\[ \dot w_i=\sum_{j=1}^{k}\frac{\partial\phi_i}{\partial w_{p_j}}\dot w_{p_j}. \]

Starting with input seed $\dot x=v$, the final tangent is $\dot y=J_f(x)v$. One seed gives one directional derivative or one Jacobian column when $v$ is a basis vector.

Reverse mode first computes primal values, initializes an output adjoint $\bar y=u$, and visits operations backward. Each operand receives

\[ \bar w_{p_j}\mathrel{+}=\bar w_i\frac{\partial\phi_i}{\partial w_{p_j}}. \]

The input adjoint is $\bar x=u^{\mathsf T}J_f(x)$. For scalar $y$, seed $u=1$ yields all input partial derivatives in one reverse accumulation. This explains the “wide versus tall” rule: reverse mode is attractive for many inputs and few outputs; forward mode for few inputs and many outputs.^[3]

AD differentiates algorithms, so mathematically equivalent programs can have different derivative numerics and computational costs. Algebraic cancellation may make the represented function constant while floating-point execution still exposes rounding pathways. Iterative solvers can be differentiated through their iterations, through an implicit equation, or through a custom rule; these choices need not agree when stopped early. A reference-grade claim names the differentiated program semantics, not only an ideal formula.

Knowledge Transfer¶

Literal transfer occurs whenever a numerical program is composed from supported differentiable primitives. The same JVP/VJP machinery applies to a neural network, an ODE solver, a ray tracer, a portfolio valuation, or a physics simulation. What changes is the primitive library, trace, active parameters, and domain validation; chain-rule accumulation remains literal.

Implementation strategies transfer too. Operator overloading is convenient for dynamic languages, source transformation can enable ahead-of-time optimization, and compiler intermediate representations can make control/data dependencies explicit. Custom derivative rules let a domain replace an inefficient or numerically poor traced derivative with a mathematically justified local linearization.

Outside numerical programs, phrases such as “differentiate the workflow automatically” are metaphors. The prime-level residues are decomposition, composition, propagation, and reverse traversal. They do not instantiate AD without a derivative-bearing program and local linear maps.

Examples¶

Scalar example. Let $f(x,y)=xy+\sin x$. A trace can be written $w_1=xy$, $w_2=\sin x$, $w_3=w_1+w_2$. At $(x,y)=(1,2)$, the primal value is $2+\sin1\approx2.8414709848$.

With forward seed $(\dot x,\dot y)=(a,b)$,

\[ \dot w_1=2a+b,\qquad \dot w_2=(\cos1)a,\qquad \dot f=(2+\cos1)a+b. \]

With reverse seed $\bar f=1$, the add sends one adjoint to each input; multiplication contributes $(y,x)=(2,1)$ and sine contributes $(\cos1,0)$. The gradient is therefore $(2+\cos1,1)\approx(2.5403023059,1)$. Both modes compose the same local rules but answer different seeded linear-map queries.

Many-parameter loss. A neural network maps millions of parameters to one minibatch loss. Reverse mode records layer activations, seeds the loss adjoint with one, and propagates VJPs backward to every parameter. This is backpropagation as a reverse-AD instance. A full dense Jacobian of all internal activations is neither required nor normally constructed.^[1]^[4]

Jacobian columns versus rows. For $f:\mathbb{R}^2\to\mathbb{R}^{1000}$, two forward basis seeds can assemble the full Jacobian by columns. Reverse mode would require up to 1000 output seeds to assemble all rows. For $g:\mathbb{R}^{1000}\to\mathbb{R}$, the comparison reverses: one reverse seed gives the gradient, whereas 1000 forward basis seeds would assemble it componentwise.^[3]

Branch boundary. For code if x >= 0: return x else: return -x, tracing at positive $x$ yields derivative $1$ and at negative $x$ yields $-1$. At $x=0$, the mathematical absolute value is not differentiable. Returning the derivative of the selected branch is an implementation convention, not proof of a derivative at the cusp.

Structural Tensions¶

Forward work versus reverse memory. Forward mode streams tangents with little tape storage but repeats across many input directions. Reverse mode amortizes a scalar gradient but retains or recomputes primal intermediates.

Trace fidelity versus mathematical intent. Differentiating executed code faithfully may expose stopping criteria, branches, clipping, and approximations that a modeler intended to idealize away. Custom rules improve intent alignment but add a new correctness obligation.

Local exactness versus global numerical stability. Every local derivative rule can be analytically correct while the composed result is inaccurate because the primal or derivative problem is ill-conditioned, overflows, underflows, or suffers cancellation.

Generality versus primitive coverage. A small differentiable core supports many programs, but external calls, sparse kernels, discrete operations, and stateful effects need explicit rules. Silent fallback is dangerous.

Compute versus storage. Checkpointing trades additional primal evaluation for a smaller reverse tape. The best schedule depends on graph depth, memory, and recomputation cost.

Mathematical derivative versus chosen generalized rule. At nonsmooth points, software conventions can be useful for optimization while differing from classical differentiability. Documentation must not erase that distinction.

Structural–Framed Character¶

Automatic Differentiation is strongly structural but computationally framed. Its skeleton—decompose into local maps, propagate a linearized effect forward or a transposed effect backward, and compose by dependency—is stable across many numerical domains. Its identity nevertheless requires mathematical derivatives, program traces, chain-rule-compatible primitives, tangent/cotangent seeds, and working arithmetic.

It therefore remains domain-specific. The cross-domain applications are all numerical computation substrates, not unrelated material systems. The portable operations are already represented by broader abstractions; AD's distinctive value is their exact realization for derivative evaluation.

Structural Core vs. Domain Accent¶

The structural core is compositional sensitivity propagation over a dependency graph. Forward mode sends effects from causes to consequences; reverse mode sends outcome sensitivity back to prerequisites. Mode selection exploits input/output dimensionality, while checkpointing manages reverse information requirements.

The domain accent is constitutive: values live in differentiable spaces, primitives expose local derivatives, composition is the chain rule, seeds are tangent or cotangent vectors, and outputs are derivatives of a numerical program. Without these roles the structure becomes generic forward/reverse propagation, not AD.

Automatic Differentiation presupposes Derivative. It is a method for evaluating the best local linear map of a differentiable computation. The proposed DAG relation is composition: AD is not a kind of derivative, because the derivative is its mathematical target and result, while AD additionally supplies program decomposition, mode, seeds, and execution semantics.

Gradient is an important result for scalar outputs, Algorithm captures executable procedure, Function (Mapping) supplies the input/output object, and Composition plus Propagation describe chain-rule flow. Approximation is a contrast rather than a parent: finite differences approximate a derivative through nonzero perturbations, while AD composes local derivatives. No additional edge is needed to state these relations.

Relationships to Other Abstractions¶

Current abstraction Automatic Differentiation Domain-specific

Parents (1) — more general patterns this builds on

Automatic Differentiation presupposes Derivative Domain-specific

Automatic Differentiation presupposes Derivative.

Hierarchy paths (2) — routes to 2 parentless roots

Automatic Differentiation → Derivative → Function (Mapping)

Show alternative path (1)

Neighborhood in Abstraction Space¶

Automatic Differentiation sits in a sparse region of the domain-specific corpus (91^st percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Unclustered & Miscellaneous (1565 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-09-08

Not to Be Confused With¶

Derivative: the mathematical local linear map that AD evaluates.
Symbolic differentiation: expression transformation and simplification.
Finite-difference differentiation: perturbation-based approximation with a step size.
Complex-step differentiation: a highly accurate perturbation method for suitable analytic programs, still not AD.
Reverse mode: one AD mode, not the entire family.
Backpropagation: reverse accumulation specialized to layered objectives such as neural-network losses.
Gradient descent: an optimizer that consumes gradients; it does not produce them by definition.
Adjoint-state method: a mathematically related sensitivity technique often implemented or interpreted through reverse mode, but not every adjoint derivation is program AD.
Symbolic–numeric code generation: may cooperate with AD but uses different transformation commitments.
Differentiable programming: the broader practice of building and optimizing differentiable software systems.
Numerical sensitivity analysis: the application question; AD is one way to compute local sensitivities.

References¶

[1] Atılım Güneş Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind, “Automatic Differentiation in Machine Learning: a Survey,” Journal of Machine Learning Research 18(153), 1–43 (2018). https://jmlr.org/papers/v18/17-468.html registry ↩a ↩b ↩c ↩d ↩e ↩f

[2] Andreas Griewank and Andrea Walther, Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, 2^nd ed., SIAM (2008), DOI 10.1137/1.9780898717761. registry ↩a ↩b ↩c

[3] JAX documentation, “Forward- and reverse-mode autodiff in JAX,” defining JVP and VJP interfaces and their input/output dimensional tradeoff. https://docs.jax.dev/en/latest/jacobian-vector-products.html registry ↩a ↩b ↩c ↩d ↩e

[4] PyTorch documentation, “Autograd mechanics” and torch.autograd, documenting its reverse-mode automatic-differentiation graph and supported numerical tensor types. https://docs.pytorch.org/docs/main/notes/autograd.html and https://docs.pytorch.org/docs/main/autograd.html registry ↩a ↩b