Stratified Benchmark Suite¶
Evaluation protocol — instantiates Problem-Distribution Fit Selection
Builds the test set as explicit per-regime strata — noise levels, subgroups, scales, scenario types — and reports each separately, so a method cannot win by acing the common cases while quietly failing the ones that matter.
A single headline score is the easiest place for a bad fit to hide: a method can post an excellent average while collapsing on a regime that the average simply outvotes. Stratified Benchmark Suite is the evaluation protocol that removes that hiding place. It samples test cases deliberately by regime — subgroup, scale, noise level, constraint level, scenario type — in proportions drawn from the problem's expected distribution, and it reports performance per stratum, never rolled up into one number. Its defining discipline is disaggregation: the suite's whole reason to exist is that the aggregate is not allowed to speak for the strata beneath it. It builds and scores the test set; it does not rank the method against alternatives, and it does not go hunting for adversarial break cases.
Example¶
A team is evaluating a speech-recognition model that reports a strong average word-error rate on its validation set. Built as a stratified suite instead of one pooled corpus, the benchmark carves the cases into strata that mirror the deployment population: native and second-language accents, quiet-office versus street-noise conditions, fast and slow speech, short commands versus long dictation, and several dialect groups. Each stratum is sized for enough utterances to give a stable per-stratum number, and results are posted stratum by stratum.
The disaggregated view tells a different story than the average did. The model is excellent on the well-represented native, low-noise strata that dominate the corpus — and its word-error rate roughly triples on one second-language accent group under street noise, a stratum large enough to matter in production but too small to move the pooled score. The average said "ship it"; the strata said "not for that population, not yet."
How it works¶
The distinguishing move is stratify-then-report, not pool-then-average. Strata are derived from the Problem Distribution Profile so the test regimes match the problems the system will actually meet; each stratum is sampled to a minimum size so its score is signal rather than noise; and the suite reports every stratum's result alongside a worst-stratum figure, refusing to collapse them into a mean that a large easy stratum would dominate. A pass is defined against a floor each stratum must clear, not against the aggregate.
Tuning parameters¶
- Stratification variables — which axes the cases are split on (subgroup, scale, noise, scenario). You only catch the failures you stratify for, so the axes chosen here decide which blind spots the suite can even see.
- Minimum samples per stratum — how many cases each stratum must hold. Too few and a stratum's score is noise masquerading as signal; too many and rare regimes are expensive or impossible to populate.
- Aggregation rule — equal-weight across strata, frequency-weighted, or worst-case. This dial sets whether the suite optimizes for the typical case or refuses to average away the weak one.
- Pass criterion — a single aggregate threshold versus a floor every stratum must clear. The floor is what gives the suite teeth against a method that trades a weak stratum for a strong one.
- Refresh linkage — whether strata are frozen or re-derived as the distribution moves; this is what keeps the suite from certifying fit against a world that has changed.
When it helps, and when it misleads¶
Its strength is exposing the subgroup and tail failures that a pooled metric structurally conceals — the territory of Simpson's paradox, where an aggregate can flatter or even reverse the pattern inside its strata[1]. It turns "the model is 94% accurate" into the far more useful "it is 94% overall and 61% on this stratum."
Its limits are equally real. The suite only catches failures on axes it thought to stratify — an unmodeled regime still slips through, which is why it complements rather than replaces adversarial probing. Thin strata produce noisy per-stratum scores that can be over-read, and a suite can be gamed by choosing convenient stratification axes that avoid the embarrassing ones — or run backwards, its strata redrawn until the favored method clears every floor. The discipline that keeps it honest is to fix the strata and their floors from the problem profile before scores are seen, and to size each stratum for a number worth trusting.
How it implements the components¶
representative_benchmark_suite— the stratified, distribution-matched set of test cases is this component: a benchmark whose composition mirrors the expected problem mix instead of an arbitrary corpus.fit_evidence_record— the per-stratum results are a durable record of disaggregated fit evidence, showing where the method holds up and where it does not.
The record here is the stratified cut; the head-to-head record of the method's margin over baselines is the Baseline Comparison Table's fit_evidence_record. This suite also does not construct adversarial break cases — that is the Challenge Case Red Team's challenge_case_suite — nor define the distribution its strata are drawn from (Problem Distribution Profile).
Related¶
- Instantiates: Problem-Distribution Fit Selection — the test set that refuses to let an aggregate score stand in for per-regime fit.
- Consumes: Problem Distribution Profile supplies the regimes and proportions the strata are built to match.
- Sibling mechanisms: Baseline Comparison Table · Challenge Case Red Team · Problem Distribution Profile · Method Bias Matrix · Regularization Path Review · Out-of-Distribution Monitor · No-Universal-Winner Claim Review · Assumption Register · Method Card or Model Card · Algorithm Portfolio Router · Benchmark Refresh Audit
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Stratified Benchmark Suite operates as an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation because it builds the test set as explicit per-regime strata — noise levels, subgroups, scales, scenario types — and reports each separately, so a method cannot win by acing the common cases while quietly failing the ones that matter.
Independent corroboration: The frozen evidence defines Stratified Benchmark Suite as 'Builds the test set as explicit per-regime strata — noise levels, subgroups, scales, scenario types — and reports each separately, so a method cannot win by acing the common cases while quietly failing the ones that matter', so its operative form is Experiment, Test & Rehearsal.
Nearest alternative: Assessment, Review & Assurance — Stratified Benchmark Suite includes features of a bounded evaluation of existing evidence or work that produces a finding or disposition, but its defining operation is an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Specialized
Rationale: A benchmark reported separately across predefined strata is statistical subgroup evaluation designed to reveal aggregate masking. NIST AI evaluation guidance requires performance measurement across relevant groups and conditions; computing supplies benchmark execution and governance supplies equity criteria.
Related originating lineages:
- Computer Science & Software Engineering — computer_science contributes computer science and software-engineering practice to this mechanism's defining operation—Builds the test set as explicit per-regime strata — noise levels, subgroups, scales, scenario types — and reports each separately, so a method cannot win by acing the common cases while quietly failing the ones that matter—without displacing the selected primary historical lineage.
- Data Science & Analytics — Data science, analytics, and operational monitoring supplies a parallel or contributing lineage for the mechanism's defining operation: builds the test set as explicit per-regime strata — noise levels, subgroups, scales, scenario types — and reports each separately, so a method cannot win by acing the common cases….
- Mathematics — Mathematical modeling, proof, and abstract-structure practice supplies a parallel or contributing lineage for the mechanism's defining operation: builds the test set as explicit per-regime strata — noise levels, subgroups, scales, scenario types — and reports each separately, so a method cannot win by acing the common cases….
- Psychology — psychology contributes experimental, clinical, and behavioral psychology to this mechanism's defining operation—Builds the test set as explicit per-regime strata — noise levels, subgroups, scales, scenario types — and reports each separately, so a method cannot win by acing the common cases while quietly failing the ones that matter—without displacing the selected primary historical lineage.
- Ethics of Technology & AI Governance — Rare groups cannot be hidden.
Review resolution: The blind reviewers disagree on primary lineage (computer_science versus statistics_experimental_design). Authoritative or primary research supports statistics_experimental_design as the best historical origin: A benchmark reported separately across predefined strata is statistical subgroup evaluation designed to reveal aggregate masking. NIST AI evaluation guidance requires performance measurement across relevant groups and conditions; computing supplies benchmark execution and governance supplies equity criteria. The cited NIST AI RMF Playbook: Measure; NIST AI 800-2, Towards a Standard for Identifying and Managing Bias directly supports the mechanism's defining operation. All independently supported contributing domains are retained without an arbitrary cap. origin_mode=cross_disciplinary_synthesis records lineage, while domain_reach=specialized records later applicability separately from provenance.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
Notes¶
The suite is where representative fit is measured, but it establishes presence of failure only on the strata it drew; it can never certify safety on regimes it did not sample. That residual — the failures no stratification anticipated — is what the Challenge Case Red Team hunts before deployment and the out-of-distribution monitor watches for after it.
References¶
[1] Simpson, E. H. "The Interpretation of Interaction in Contingency Tables". Journal of the Royal Statistical Society: Series B 13(2), 238–241 (1951). Shows that aggregation can conceal subgroup patterns and even reverse an association visible within the strata. registry ↩