Automatic Differentiation

A family of program-evaluation and transformation techniques that decomposes an executed numerical computation into differentiable primitives and composes their local derivative rules to obtain derivatives accurate to working precision, chiefly through forward Jacobian–vector or reverse vector–Jacobian accumulation.

Version

v1 · 2026-08-30 · History

Domain-specific #

1322

Origin domain

numerical computing

Subdomain

derivative computation

Aliases

Algorithmic differentiation, Autodiff, Auto-differentiation, AD

Core Idea

Automatic differentiation (AD) is a family of techniques for evaluating derivatives of numerical functions represented by computer programs. It observes that an executed computation is a composition of elementary operations—addition, multiplication, division, trigonometric functions, exponentials, linear-algebra primitives, and other operations supplied with derivative rules. AD applies the chain rule to those operations in the order required by the program, producing derivatives of the represented computation without first converting the whole program into a symbolic formula and without perturbing inputs by a finite step.

Scope of Application

AD recurs across scientific computing, nonlinear optimization, optimal control, computational physics, inverse problems, statistics, computer graphics, robotics, finance, and machine learning. These practices share a need for gradients, Jacobians, Hessian products, and parameter sensitivities of calculations too large or changeable for manual derivatives.

In machine learning, reverse mode differentiates a scalar loss with respect to many parameters. In simulation and optimization, forward mode can propagate selected parameter or state directions, while reverse/adjoint mode can obtain sensitivities of a small number of objectives to many inputs.

Clarity

A claim of AD should specify five things:

1. Primal function and active inputs. Which arguments are differentiated, at which values, and which are treated as constants? 2. Mode and seed. Is the result a JVP \(Jv\), VJP \(u^{\mathsf T}J\), scalar gradient, dense Jacobian, or nested higher derivative? 3. Trace semantics. Is the graph static, traced dynamically, transformed from source, or built by operator overloading?

Manages Complexity

The key compression is reuse of the primal program's decomposition. Instead of deriving and maintaining one monolithic formula, an AD system maintains local rules for a comparatively small primitive set. Every program assembled from those primitives inherits derivative evaluation through the chain rule. When the primal code changes, derivative behavior changes with the trace, reducing manual synchronization errors.

Abstract Reasoning

For an operation \(w_i=\phi_i(w_{p_1},\ldots,w_{p_k})\), forward mode propagates a tangent

\[ \dot w_i=\sum_{j=1}^{k}\frac{\partial\phi_i}{\partial w_{p_j}}\dot w_{p_j}. \]

Starting with input seed \(\dot x=v\), the final tangent is \(\dot y=J_f(x)v\). One seed gives one directional derivative or one Jacobian column when \(v\) is a basis vector.

Knowledge Transfer

Literal transfer occurs whenever a numerical program is composed from supported differentiable primitives. The same JVP/VJP machinery applies to a neural network, an ODE solver, a ray tracer, a portfolio valuation, or a physics simulation. What changes is the primitive library, trace, active parameters, and domain validation; chain-rule accumulation remains literal.

Implementation strategies transfer too. Operator overloading is convenient for dynamic languages, source transformation can enable ahead-of-time optimization, and compiler intermediate representations can make control/data dependencies explicit.

Relationships to Other Abstractions

Current abstraction Automatic Differentiation Domain-specific

Parents (1) — more general patterns this builds on

• Automatic Differentiation presupposes Derivative Domain-specific

  Automatic Differentiation presupposes Derivative.

Hierarchy paths (2) — routes to 2 parentless roots

• Automatic Differentiation → Derivative → Function (Mapping)

Show alternative path (1)

Neighborhood in Abstraction Space

Automatic Differentiation sits in a sparse region of the domain-specific corpus (91^st percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Unclustered & Miscellaneous (1565 abstractions)

Nearest neighbors

• S-m-n Theorem — 0.80
• Symbolic Execution — 0.78
• Zémor's Decoding Algorithm — 0.78
• Function-Level Programming — 0.77
• Formula Calculator — 0.77

Computed from structural-signature embeddings · 2026-09-08