Large width limits of neural networks¶

Asymptotic regimes in which neural-network layer widths tend to infinity and random networks converge to analytically tractable Gaussian-process, kernel, mean-field, or feature-learning descriptions.

Version: v2 · 2026-09-19 · History
Domain-specific #: 5263
Domain group: Applied Sciences & Engineering
Origin domain: Computer Science & Software Engineering
Subdomain: Theoretical Machine Learning → Computer Science & Software Engineering

Core Idea¶

Large-width limits replace a finite interacting parameter system with a limiting stochastic process or deterministic evolution, but the result depends on architecture, initialization and learning-rate scaling, training rule, depth, and order of limits. Width-dependent random sums concentrate or satisfy central-limit behavior under a declared parameterization; training dynamics then converge to a kernel, feature distribution, or mean-field equation on a specified time scale. The abstraction is therefore identified by a declared carrier, a transformation or constraint over that carrier, and an invariant that tells an analyst whether the named structure is genuinely present.

How would you explain it like I'm…

Super-Wide Robot Brains

A computer brain is made of many tiny pieces that each push a little. If you make it super wide, with tons of pieces, all the little pushes add up to something smooth and predictable. But which predictable thing you get depends on how you set it up and how you teach it.

Infinitely Wide Networks

Neural networks are computer programs built from layers of many small units. When scientists imagine the layers getting wider and wider without end, the messy network starts to act like a simpler, predictable process they can study with math. The surprise is that there isn't just one answer. The limit you reach depends on how you start the weights, how big the learning steps are, how deep the network is, and even which thing you let grow first. So you have to spell out all of those choices before you can say what happens.

Parameterization-Dependent Width Limits

Large width limits study what a neural network turns into as the number of neurons per layer goes to infinity. Sums over many random weights either settle to a fixed value or behave like a Gaussian, much as the central limit theorem predicts, and training can then be described by a simpler limiting object. That object might be a fixed kernel, an evolving distribution of features, or a mean-field equation. The limit you get is not universal: it depends on the architecture, how the initial weights are scaled with width, how the learning rate is scaled, the training rule, the depth, and which limit is taken first. A claim about wide networks is only meaningful when all these choices, plus how far a finite network is from the limit, are stated explicitly.

Large-width limits replace a finite network, a system of many interacting parameters, with a limiting stochastic process or deterministic evolution as width n → ∞. Under a declared parameterization, random sums over width concentrate (law of large numbers) or have Gaussian fluctuations (central limit behavior). At initialization this can give Gaussian-process behavior. Under training, the dynamics may converge to kernel dynamics, a feature-learning distribution, or a mean-field PDE, each on a specified time scale. Which limit appears depends on the architecture sequence, the initialization law and its variance scaling, the learning-rate scaling, the optimizer, the depth regime, the convergence mode, and the order in which width, depth, and time limits are taken. Finite-width error must also be quantified. The concept is therefore not 'big networks' in general. It is a precisely specified limit, and conclusions should not be carried over to settings that violate its assumptions.

Scope of Application¶

Large width limits of neural networks belongs to theoretical machine learning and is useful where the analyst can specify the typed theoretical machine learning carrier, defining objects and relations, parameters, conventions, evidence, boundary cases, and comparison targets, then evaluate the architecture sequence, width parameter, initialization law and variance scaling, depth regime, optimizer and learning-rate scaling, convergence mode, order of limits, and finite-width error are explicit. The scope is broad within that domain but bounded by the need for the architecture sequence, width parameter, initialization law and variance scaling, depth regime, optimizer and learning-rate scaling, convergence mode, order of limits, and finite-width error are explicit.

Clarity¶

The abstraction clarifies a crowded vocabulary by making the architecture sequence, width parameter, initialization law and variance scaling, depth regime, optimizer and learning-rate scaling, convergence mode, order of limits, and finite-width error are explicit the center of the account. A claim should name the carrier, the governing operation or relation, the applicable assumptions, and the recognition test.

Manages Complexity¶

Without the abstraction, an analyst must reason directly over many local details: the carrier roles, admissibility assumptions, competing conventions, derived invariants, boundary cases, and proof or validation obligations specific to Large width limits of neural networks. Large width limits of neural networks compresses them into the roles in the structural signature. That compression permits comparison across instances without erasing the variables that determine validity. It also exposes which details may be varied safely and which are constitutive.

Abstract Reasoning¶

Identify the carrier. State what the elements, states, objects, or observations are: the typed theoretical machine learning carrier, defining objects and relations, parameters, conventions, evidence, boundary cases, and comparison targets. Reject examples whose alleged carrier belongs to a different problem.
Lock the constitutive rule. Express the architecture sequence, width parameter, initialization law and variance scaling, depth regime, optimizer and learning-rate scaling, convergence mode, order of limits, and finite-width error are explicit independently of one notation or implementation.

Knowledge Transfer¶

Knowledge transfers strongly among subfields of theoretical machine learning because they reuse the typed theoretical machine learning carrier, defining objects and relations, parameters, conventions, evidence, boundary cases, and comparison targets, Width-dependent random sums concentrate or satisfy central-limit behavior under a declared parameterization; training dynamics then converge to a kernel, feature distribution, or mean-field equation on a specified time scale., and type the carrier, state every parameter and convention in the definition, test that the architecture sequence, width parameter, initialization law and variance scaling, depth regime, optimizer and learning-rate scaling, convergence mode, order of limits, and finite-width error are explicit, compare the nearest accepted identity, and report counterexamples, uncertainty, and limiting cases.

Relationships to Other Abstractions¶

Current abstraction Large width limits of neural networks Domain-specific

Parents (1) — outgoing typed relations

Large width limits of neural networks is a kind of Scaling and Scale Dependence Prime

It remains its own entry because its identity is fixed by the architecture sequence, width parameter, initialization law and variance scaling, depth regime, optimizer and learning-rate scaling, convergence mode, order of limits, and finite-width error.

Hierarchy path (1) — routes to 1 parentless root

Large width limits of neural networks → is a kind of Scaling and Scale Dependence → is a decomposition of Scale

Neighborhood in Abstraction Space¶

Large width limits of neural networks sits in a crowded region of the domain-specific corpus (13^th percentile for distinctiveness): several abstractions share nearly its structure, so a description that fits it tends to fit its neighbors too.

Family — Machine Learning Models & Representations (25 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-10-25