Neural scaling law¶
An empirical relation describing how neural-network loss or capability changes with model size, data, training compute or inference compute.
Core Idea¶
Fits are regime- and architecture-dependent, extrapolation can fail at data quality or optimization transitions and benchmark contamination or selective reporting can mimic smooth scaling. Controlled training or evaluation runs vary one or jointly allocated resources, measured loss is fit to a power law or related curve and the fitted exponents predict marginal returns and compute-optimal allocation within a validated range. The abstraction is therefore identified by a declared carrier, a transformation or constraint over that carrier, and an invariant that tells an analyst whether the named structure is genuinely present.
Scope of Application¶
Neural scaling law belongs to machine learning and is useful where the analyst can specify the typed machine learning carrier, including objects, relations, parameters, conventions, evidence, boundaries, and comparison targets, then evaluate the model family and training recipe, parameter count data tokens training and inference compute, outcome metric and evaluation set, scaling variables held fixed or co-varied, fitted functional form exponents and uncertainty, observed range and extrapolation limit and compute-optimal allocation rule are explicit.
Clarity¶
The abstraction clarifies a crowded vocabulary by making the model family and training recipe, parameter count data tokens training and inference compute, outcome metric and evaluation set, scaling variables held fixed or co-varied, fitted functional form exponents and uncertainty, observed range and extrapolation limit and compute-optimal allocation rule are explicit the center of the account. A claim should name the carrier, the governing operation or relation, the applicable assumptions, and the recognition test.
Manages Complexity¶
Without the abstraction, an analyst must reason directly over many local details: the carrier roles, admissibility assumptions, competing conventions, derived invariants, boundary cases, and proof or validation obligations specific to Neural scaling law. Neural scaling law compresses them into the roles in the structural signature. That compression permits comparison across instances without erasing the variables that determine validity. It also exposes which details may be varied safely and which are constitutive.
Abstract Reasoning¶
- Identify the carrier. State what the elements, states, objects, or observations are: the typed machine learning carrier, including objects, relations, parameters, conventions, evidence, boundaries, and comparison targets. Reject examples whose alleged carrier belongs to a different problem. 2. Lock the constitutive rule. Express the model family and training recipe, parameter count data tokens training and inference compute, outcome metric and evaluation set, scaling variables held fixed or co-varied, fitted functional form exponents and uncertainty, observed range and extrapolation limit and compute-optimal allocation rule are explicit independently of one notation or implementation.
Knowledge Transfer¶
Knowledge transfers strongly among subfields of machine learning because they reuse the typed machine learning carrier, including objects, relations, parameters, conventions, evidence, boundaries, and comparison targets, Controlled training or evaluation runs vary one or jointly allocated resources, measured loss is fit to a power law or related curve and the fitted exponents predict marginal returns and compute-optimal allocation within a validated range., and type the carrier, state every parameter and convention in the definition, test that the model family and training recipe, parameter count data tokens training and inference compute, outcome metric and evaluation set, scaling variables held fixed or co-varied, fitted functional form exponents and uncertainty, observed range and extrapolation limit and compute-optimal allocation rule are explicit, compare the nearest accepted identity, and report counterexamples, uncertainty, and limiting cases.
Relationships to Other Abstractions¶
Current abstraction Neural scaling law Domain-specific
Parents (1) — more general patterns this builds on
-
Neural scaling law is a kind of Proportionality Prime
The proposed strict upward parent is
prime:proportionality.
Hierarchy paths (2) — routes to 2 parentless roots
- Neural scaling law → Proportionality → Normativity → Constraint
- Neural scaling law → Proportionality → Commensurability
Neighborhood in Abstraction Space¶
Neural scaling law sits in a crowded region of the domain-specific corpus (19th percentile for distinctiveness): several abstractions share nearly its structure, so a description that fits it tends to fit its neighbors too.
Family — Deep Learning Architectures & Scaling (16 abstractions)
Nearest neighbors
- Large width limits of neural networks — 0.94
- Neural Turing machine — 0.92
- Lazy learning — 0.92
- Model compression — 0.91
- Ensemble learning — 0.91
Computed from structural-signature embeddings · 2026-09-08