Exact-or-Numerical Benchmark¶
Validation reference — instantiates Resummation and Nonperturbative Extrapolation
Supplies independent exact values and high-fidelity numerical results, held out from tuning, to test a reconstruction and cross-check competing methods near the target.
A resummation can be internally consistent, stable across orders, and still wrong. Exact-or-Numerical Benchmark is the mechanism that supplies the outside evidence — exactly solvable limits, and expensive high-fidelity numerical results — against which any reconstruction is tested. It builds nothing itself; its entire value is independence. The defining discipline is the hold-out: a benchmark case that was used to tune a transform can no longer certify it, so the mechanism partitions its cases into those that fit methods and those reserved to judge them, and it deliberately stocks the hard cases that sit near the target regime rather than the easy ones near the expansion origin where every method reproduces the input. Its output is not a number but a verdict about numbers: which reconstructions survive contact with evidence they were not fitted to.
Example¶
A group extrapolating a quantity in a lattice field theory has three candidate reconstructions from three different transform families, all of which agree with each other to a few percent. Before trusting that agreement, they consult independent Monte Carlo simulations run at several coupling values — expensive, but exact up to controlled statistical error. Crucially, two of the simulation points are held out from every method's tuning. The reconstructions are compared not only to each other but to those held-out points: two of the three land within the simulation's error bars near the target coupling, while the third — which had looked equally good on the easy near-origin cases — drifts off by a clear margin as the coupling grows. The mechanism's report is a validity envelope: the two surviving reconstructions are certified over the tested coupling range, the drifting one is set aside, and a note records that a new simulation point deeper into the target regime would reopen the judgment.
How it works¶
- Assemble the reference cases. Gather exact limits and high-fidelity numerics, prioritizing cases near the target rather than the easy origin.
- Partition tuning from judgment. Reserve some cases as strictly held out; when data are scarce, rotate leave-one-out so no case both trains and grades.
- Test approach, not just origin. Score each reconstruction on how it tracks the benchmarks toward the target, where methods diverge.
- Rule on consensus and validity. Certify agreement only when methods match the held-out cases and one another, and record what new benchmark would reopen the verdict.
Tuning parameters¶
- Hold-out fraction — how many cases are withheld from all tuning. More held-out cases give a stronger test but leave fewer to constrain methods.
- Benchmark proximity — how close the cases sit to the target regime. Distant, hard cases test what matters; near-origin cases flatter everyone.
- Agreement tolerance — how tight a match to a benchmark counts as a pass. Too loose certifies drift; too tight rejects on benchmark noise.
- Rotation depth — with scarce data, how many leave-one-out folds are run before the residual is reported (and never as an out-of-sample guarantee).
When it helps, and when it misleads¶
Its strength is that it converts mutual agreement — which can merely reflect shared assumptions — into tested agreement, and it localizes the next expensive calculation to where methods disagree most. A held-out exact or numerical case is the one piece of evidence a resummation cannot argue with.
Its failure mode is benchmark leakage[1]: using the same case to tune a method and then to certify it, which reports an in-sample fit as if it were out-of-sample validation and quietly guarantees a pass. The classic misuse is selecting the Padé shape that best matches a known value and presenting that match as confirmation. The guarding discipline is strict partitioning — held-out cases never touch tuning — and, when data are too scarce to hold out, disclosing that the reported error is in-sample and widening rather than narrowing the stated uncertainty.
How it implements the components¶
benchmark_case_set— its core artifact: curated exact and high-fidelity numerical cases, some strictly held out, weighted toward the target regime.cross_method_consensus_rule— certifies agreement only when independent reconstructions match the held-out cases and each other, not merely one another.validity_and_reopening_rule— sets the range over which reconstructions are certified by the benchmarks and names the new case that would reopen it.
It reconstructs nothing of its own — the resummation_transform_choice and continuation_ensemble belong to method siblings such as Borel Resummation and Padé Approximant — and it tests general reconstructions rather than binding one to a single strong-regime exact_limit_anchor, which is Strong-Coupling Extrapolation Check's job.
Related¶
- Instantiates: Resummation and Nonperturbative Extrapolation — the independent-evidence layer that certifies or rejects reconstructions.
- Sibling mechanisms: Borel–Padé Resummation · Borel Resummation · Conformal Borel Mapping · High-Temperature Series Resummation · Matched Asymptotic Expansion · Padé Approximant · Renormalization-Group Improvement · Sequence Acceleration Transform · Strong-Coupling Extrapolation Check
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Exact-or-Numerical Benchmark operates as a bounded trial, probe, simulation, or rehearsal that generates evidence from performance because it supplies independent exact values and high-fidelity numerical results, held out from tuning, to test a reconstruction and cross-check competing methods near the target.
Independent corroboration: The frozen evidence defines Exact-or-Numerical Benchmark as 'Supplies independent exact values and high-fidelity numerical results, held out from tuning, to test a reconstruction and cross-check competing methods near the target', so its operative form is Experiment, Test & Rehearsal.
Nearest alternative: Assessment, Review & Assurance — Held-out exact and numerical cases actively test reconstruction behavior near the target, beyond reviewing an existing validation report.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Physics
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Specialized
Rationale: Benchmarking analytic reconstructions against exactly solvable limits and high-fidelity numerics is characteristic theoretical and computational physics practice.
Related originating lineages:
- Computer Science & Software Engineering — Held-out evaluation and numerical-comparison practice materially shape its anti-leakage protocol. Held-out numerical reference implementations and computational validation materially shape the executable benchmark form.
- Mathematics — Exact solutions and convergence theory materially supply the reference side of the benchmark. Exact solutions and convergence analysis provide an independent formal benchmark lineage.
Review resolution: Both reviewers agree that physics is primary. I retain computer_science, mathematics only as formative origin lineages; cross_disciplinary_synthesis is appropriate because the final form materially combines the agreed primary with the retained formative lineages. Reach is specialized because the mechanism remains tied to a bounded professional technique, an applicability judgment kept separate from provenance. Encyclopedia synthesis is true because the exact generalized artifact is an encyclopedia-authored combination or refinement. No unresolved historical ambiguity remains after reconciling the secondary fields.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
The two validation-flavored siblings divide cleanly. This mechanism is a suite: many independent cases, held out, testing whichever reconstructions are on the table. Strong-Coupling Extrapolation Check is a single-limit test: it binds one reconstruction to one known exact behavior in a specific regime. Keeping the suite separate is what lets the same held-out cases judge every method impartially.
References¶
[1] Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. "Leakage in Data Mining: Formulation, Detection, and Avoidance". ACM Transactions on Knowledge Discovery from Data 6(4), Article 15, 1–21 (2012). Defines benchmark leakage as using evaluation information in model or parameter selection, producing an in-sample estimate that overstates out-of-sample performance. registry ↩