Benchmark Harness¶
Measurement tool / benchmark rig — instantiates Equivalence-Preserving Rewrite Optimization
Measures the orthogonal cost of a rewrite — speed, memory, size — under controlled, repeatable conditions, so a "faster" form can be shown faster rather than assumed.
Once two forms are known to be equivalent, the remaining question is which one is actually better, and the honest answer to that is a measurement, not a hunch. The Benchmark Harness is the rig that measures it: it runs the original and the rewritten form under matched, repeatable conditions and reports the cost — latency, throughput, memory, code size — that the rewrite was supposed to improve. Its defining discipline is measurement rigor: warm-up to steady state, many repetitions, controlled inputs, and a reported spread, so that a real gain is separated from noise. Crucially, it answers only the cost half of the archetype and never the correctness half — it assumes equivalence has already been established and refuses to conflate "faster" with "right."
Example¶
An engineer replaces a hot inner routine — say a checksum over network packets — with an algebraically equivalent form that trades a division for a reciprocal-multiply, and claims "about 2× faster." The Harness is where that claim is tested. It runs both versions after a warm-up phase (to reach cache- and JIT-steady state), repeats each ≈10,000 times across inputs of varied size, randomizes the order to cancel drift, and reports the distribution: median latency plus the interquartile range.
The outcome refines the story rather than rubber-stamping it. On small packets the two forms are within noise — the "speed-up" vanishes. On large packets the rewrite is ≈1.4× faster, not 2×, and peak memory ticked up slightly. That is enough to change the decision: accept the rewrite on the large-packet path, leave the small-packet path alone. The number the engineer felt becomes a number the team can act on.
How it works¶
The Harness runs strictly after an equivalence check has passed — it is a measurer, not an oracle, and it takes the forms' sameness as given. Its whole value is in suppressing confounders that make naive timing lie: it warms up before measuring, repeats to expose variance, fixes the input distribution, isolates the environment, and reports a spread (median and tail) rather than a single flattering number. It also attributes the cost to a specific criterion — latency is not throughput is not memory is not binary size — because a rewrite that wins on one routinely loses on another, and a single "score" hides that.
Tuning parameters¶
- Cost criterion — which axis is measured: latency, throughput, peak memory, code size, energy. Optimizing one can silently worsen another; the harness makes you name the one that matters.
- Workload / input distribution — representative versus worst-case inputs. A rewrite that wins on the average case can lose badly on the tail, so the chosen distribution decides what "faster" even means.
- Repetition and warm-up — how many iterations and how much warm-up. More reduces variance and reaches steady state but costs wall-clock time.
- Reported statistic — median, mean, or p99. The choice exposes or hides tail behaviour; a mean can be dragged around by a few outliers.
- Environment fidelity — an isolated microbenchmark versus a production-like setting. Isolation shrinks noise but can report a local win that never materializes in the whole system.
When it helps, and when it misleads¶
Its strength is that it turns the cost side of the archetype empirical: it kills "it feels faster" folklore and gives the acceptance decision a measured, reproducible basis with its uncertainty attached. It also cleanly separates measuring a rewrite from accepting it — a win inside the noise band is no win at all.
Its central trap is the microbenchmark that does not reflect the whole system: a dramatic local speed-up on a routine that is a tiny fraction of total runtime barely moves anything, because the achievable whole-system gain is bounded by the fraction of time actually spent there.[n1] It is also vulnerable to measurement bias — unrepresentative inputs, a cold cache, or a compiler that optimizes away a benchmark whose result is unused — and to the classic misuse of running it after a decision, on the one workload that flatters the rewrite already shipped. The discipline that keeps it honest is to pre-declare the workload, the criterion, and the reported statistic before measuring, to always report the spread, and to confirm the local win at the whole-system level.
How it implements the components¶
The Harness fills the archetype's measurement components — the cost side, held strictly orthogonal to correctness:
benchmark_or_cost_measurement_panel— it is the instrument: the rig that runs the forms and reports their cost with controlled variance.orthogonal_cost_criterion— it defines and quantifies the specific axis being optimized (speed, memory, size), the "better" that the rewrite is judged against, kept separate from whether the rewrite is correct.
It does not establish that the forms are equivalent in the first place (Golden-Output Regression Test, Metamorphic Test Suite), and it does not generate or choose the rewrite (Compiler Optimization Pass) — it only measures what those produce.
Related¶
- Instantiates: Equivalence-Preserving Rewrite Optimization — the Harness supplies the empirical cost input the acceptance decision depends on.
- Consumes: an equivalence verdict — it measures only after a pass from Golden-Output Regression Test or Metamorphic Test Suite.
- Sibling mechanisms: Compiler Optimization Pass · Golden-Output Regression Test · Metamorphic Test Suite · Property-Based Equivalence Test · Peephole Optimization
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Measures the orthogonal cost of a rewrite — speed, memory, size — under controlled, repeatable conditions, so a 'faster' form can be shown faster rather than assumed, making its operative form a deliberate probe, variation, simulation, or practiced execution used to generate evidence or readiness.
Independent corroboration: The frozen evidence defines Benchmark Harness as 'Measures the orthogonal cost of a rewrite — speed, memory, size — under controlled, repeatable conditions, so a 'faster' form can be shown faster rather than assumed', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Single lineage
Present-day reach: Specialized
Rationale: Software performance engineering uses controlled harnesses, warm-up, repeated trials, fixed workloads, and spread statistics to compare equivalent implementations.
Related originating lineages:
- Engineering & Design — Verification practice contributes controlled conditions, tolerances, and calibrated instrumentation.
- Statistics & Experimental Design — Repeated trials, variance control, and uncertainty reporting keep timing comparisons honest.
Review resolution: Computer science is the agreed primary lineage through controlled performance benchmarking of code. Statistics supplies repeatability and uncertainty, while engineering verification supplies environmental control and tolerances.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Amdahl's law: the overall speed-up from improving one part of a system is capped by the fraction of total time that part consumes. It is the standard reason a genuine, well-measured local win can still be worthless at the system level — which is why the Harness's results must be checked against whole-system impact. ↩