Scale-Sweep Benchmark¶
Test or assessment — instantiates Dominant-Term Regime Modeling
A benchmark or simulation across multiple scales used to detect whether predicted dominance appears.
Scale-Sweep Benchmark is an empirical experiment: it runs the real system at a deliberately chosen set of scales and watches to see whether the term you predicted would dominate actually takes over inside your operating range. Its whole reason for existing is that an asymptotic prediction can be mathematically impeccable and still be wrong in practice — because the crossover it implies sits far beyond any scale you will ever reach, or because a hidden constant reverses the ranking long before then. The benchmark refuses to trust the prediction on paper. It generates measurements by execution, so where an analytic tool derives dominance and a plot reads it, this one witnesses it — or fails to. The output is a verdict grounded in observed behavior: the predicted regime either shows up within the swept scales, or it does not.
Example¶
An ML infrastructure team predicts that at large batch sizes model inference will be limited by memory bandwidth, while at small batch sizes fixed per-call kernel-launch overhead dominates. Rather than trust the model, they define a scenario set of batch sizes — 1, 8, 64, 512, 4096 — and measure end-to-end latency at each, holding the model, hardware, and precision fixed. The measurements tell the story the analysis could not confirm on its own: overhead does dominate below a batch size of roughly thirty, bandwidth dominates above it, and the crossover sits inside their real serving range rather than off in some unreachable regime. The prediction is validated by execution, and the serving batch size is chosen on evidence rather than on a formula.
How it works¶
- Define a scenario set of scales spanning — and extending past — the decision-relevant range.
- Execute the system at each scale, holding all other conditions fixed so the scale variable is the only thing moving.
- Look for the predicted dominant term to assert itself, and note the scale at which it takes over.
- Confirm the candidate terms behave as ranked; a ranking that never materializes falsifies the prediction.
Tuning parameters¶
- Sweep range and spacing — geometric steps cover many orders cheaply; too narrow a range can stop short of the crossover and miss it entirely.
- Replication and noise control — repeats and warmup reduce measurement noise at the cost of runtime.
- Fixed-condition set — what is held constant; freezing the wrong variable can hide a co-dominant driver.
- Overshoot — how far past the decision range the sweep extends, buying confidence that the regime is truly entered.
When it helps, and when it misleads¶
Its strength is catching premature asymptotics and constant-factor reversals that pure analysis waves away: it is the only mechanism here that can show a predicted dominance simply failing to appear. Amdahl's law[n1] describes exactly the kind of dominance a sweep is built to watch emerge — the serial fraction quietly taking over as parallel scale grows.
Its failure mode is a sweep too narrow to reach the crossover, which then "confirms" whichever term happens to lead in the sampled band — and the closely related trap of overfitting to the one benchmarked scenario. The guarding discipline is to extend the sweep until the predicted regime is genuinely entered, and to vary the fixed conditions so the result is not an artifact of a single configuration.
How it implements the components¶
scale_scenario_set— it defines and runs the explicit set of scales; choosing that set is the experiment's design.crossover_validation_probe— measuring at each scale is precisely the probe for whether, and where, the predicted dominance appears in range.dominant_term_candidate_set— it exercises the enumerated candidate terms empirically, letting each reveal its real growth.
It does not read a growth exponent off a slope of already-collected data — that is Log-Log Scaling Plot, its nearest twin: the plot displays measurements, the benchmark produces them by running the system; nor does it derive an analytic crossover — that is Crossover-Point Calculation.
Related¶
- Instantiates: Dominant-Term Regime Modeling — the empirical check that a predicted regime actually arrives.
- Consumes: Big-O / Landau Notation or Ratio Limit Test — supplies the predicted ranking the sweep sets out to confirm or falsify.
- Sibling mechanisms: Big-O / Landau Notation · Ratio Limit Test · Log-Log Scaling Plot · Dominant Balance Table · Crossover-Point Calculation · Finite-Size Correction Check · Asymptotic Claim Review
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Scale-Sweep Benchmark operates as an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation because it a benchmark or simulation across multiple scales used to detect whether predicted dominance appears.
Independent corroboration: The frozen evidence defines Scale-Sweep Benchmark as 'A benchmark or simulation across multiple scales used to detect whether predicted dominance appears', so its operative form is Experiment, Test & Rehearsal.
Nearest alternative: Analysis, Modeling & Optimization — Scale-Sweep Benchmark includes features of an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution, but its defining operation is an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Engineering & Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Benchmarking the same system across systematically increasing scales to reveal dominance changes is engineering performance characterization. NIST benchmarking and screening guidance varies controlled factors, holds inputs comparable, and analyzes performance distributions; statistics supplies inference from the sweep.
Related originating lineages:
- Computer Science & Software Engineering — computer_science contributes algorithms, data models, release isolation, and executable procedures to the mechanism's formative or independently convergent form; that contribution does not displace the primary engineering_design lineage.
- Data Science & Analytics — Data science, analytics, and operational monitoring supplies a parallel or contributing lineage for the mechanism's defining operation: a benchmark or simulation across multiple scales used to detect whether predicted dominance appears.
- Mathematics — Mathematical modeling, proof, and abstract-structure practice supplies a parallel or contributing lineage for the mechanism's defining operation: a benchmark or simulation across multiple scales used to detect whether predicted dominance appears.
- Physics — physics contributes signal, energy-flow, dynamical, and stability analysis to the mechanism's formative or independently convergent form; that contribution does not displace the primary engineering_design lineage.
- Statistics & Experimental Design — Scale-Sweep Benchmark's terminology and operating form—a benchmark or simulation across multiple scales used to detect whether predicted dominance appears—are rooted most directly in statistics, experimental design, and measurement theory.
Review resolution: The blind reviewers disagreed on primary lineage (engineering_design versus statistics_experimental_design); authoritative or primary research supports engineering_design as the best historical origin. Benchmarking the same system across systematically increasing scales to reveal dominance changes is engineering performance characterization. NIST benchmarking and screening guidance varies controlled factors, holds inputs comparable, and analyzes performance distributions; statistics supplies inference from the sweep. The cited NIST, Screening Factors Affecting Application Performance; NIST, PFHub Benchmark 7 directly supports the defining operation used in that choice. All independently supported contributing domains are retained without an arbitrary cap, while domain_reach=multi_domain records later applicability separately from provenance.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
Notes¶
[n1] Amdahl's law states that the speedup of a parallelized task is ultimately capped by the fraction that must run serially — so as scale grows, the serial part comes to dominate the runtime. It is a canonical example of a dominant term whose takeover a scaling sweep can observe directly. ↩