Baseline Comparison Table¶
Evaluation artifact — instantiates Problem-Distribution Fit Selection
Scores the candidate method head-to-head against a deliberately assembled ladder of reference points — trivial, incumbent, simple-but-strong, robust, domain-specific, and human-assisted — under identical conditions, so an apparent win has to survive comparison with what it claims to beat.
A method's raw score means almost nothing on its own; what matters is its margin over the right alternative. Baseline Comparison Table supplies that alternative — or rather a whole ladder of them. It scores the candidate against a curated set of reference points on the same tasks, the same metric, and the same budget: a trivial rule, the current incumbent, a tuned simple baseline, a robust general method, a domain-specific tool, and a human-in-the-loop option. Where the Method Bias Matrix reasons about which method should fit, this artifact measures whether it actually does, and against what. Its defining discipline is that "state of the art" is not a number but a delta: the candidate must clear the height of what already exists, and the table is the record of by how much.
Example¶
A payments processor is pitched a deep-learning fraud model that "detects ≈30% more fraud." Before adopting it, the risk team builds the comparison table. The reference ladder: "flag every transaction over $500" (trivial), the existing hand-written rules engine (incumbent), a tuned logistic regression on the same features (simple-but-strong), a random forest (robust), a velocity-and-geography heuristic (domain-specific), and analyst-in-the-loop review of flagged cases (human-assisted). Every row is scored on the same held-out month, on the same cost metric that trades fraud caught against false-positive friction, with each baseline tuned by someone who wants it to win.
The filled table reframes everything. The deep model does top raw detection — but the tuned logistic regression captures most of that lift at a fraction of the cost and with full interpretability, and the incumbent rules already handle the single largest fraud class for free. The decision is no longer "is the model good?" but "is the incremental lift over logistic regression worth the opacity and the retraining pipeline?" — a question the headline "30% more" had hidden entirely.
How it works¶
The distinguishing element is the deliberately built baseline ladder scored under held-identical conditions. You assemble references from trivial up through human-assisted so the candidate is bracketed from below by things that are cheap and from the side by things that are already trusted; you run every entry on the same tasks, metric, and budget so no row enjoys a hidden advantage; and you report absolute performance next to the increment over each baseline, with cost, latency, and interpretability as their own columns so "better" is forced to be multi-dimensional. The candidate's number is meaningless alone and only acquires meaning as a margin.
Tuning parameters¶
- Baseline ladder composition — which reference points are admitted. Omitting a strong simple baseline is the single easiest way to manufacture a win, so the ladder's honesty is set here more than anywhere else.
- Shared-condition strictness — how rigidly tasks, metric, and budget are held identical across rows. Any looseness lets the candidate's hidden advantages leak into the margin.
- Metric-set width — accuracy alone versus accuracy plus cost, latency, and interpretability. Narrow columns systematically flatter the more complex method.
- Tuning parity — how much optimization effort each baseline receives. An untuned baseline is a strawman, and the delta over a strawman is fiction.
When it helps, and when it misleads¶
Its strength is deflating inflated wins into honest increments: it quantifies exactly how much the candidate adds over what you could already do, and it repeatedly reveals that a simpler, cheaper option is good enough — the outcome benchmark theater is built to hide.
Its limits are equally sharp. The table only measures on the tasks inside it, so a method that wins here can still fail out of distribution — that blind spot belongs to the Challenge Case Red Team and the out-of-distribution monitor, not here.[1] And a single weak or untuned baseline silently inflates every margin above it. Its classic misuse is choosing a deliberately feeble baseline so the candidate shines — or, run backwards, adding and dropping baselines until the pre-chosen favorite comes out on top. The guardrail is to pre-register the baseline ladder and the metric before any scores are seen, and to have each baseline tuned by someone motivated to make it win.
How it implements the components¶
baseline_and_alternative_set— the deliberately assembled ladder of trivial, incumbent, simple, robust, domain-specific, and human-assisted references is this component, made concrete and scoped to the decision.fit_evidence_record— the filled table is the durable record of comparative fit evidence: measured performance of every option on identical tasks, with margins and costs attached.
It does not profile any method's internal method_bias_profile — that is the Method Bias Matrix's a-priori job — and it does not probe the challenge_case_suite of out-of-distribution failures; the Challenge Case Red Team supplies those.
Related¶
- Instantiates: Problem-Distribution Fit Selection — the empirical head-to-head that turns a candidate's score into a defensible margin.
- Consumes: Stratified Benchmark Suite supplies the tasks every row is scored on; the Method Bias Matrix supplies the shortlist worth scoring.
- Sibling mechanisms: Method Bias Matrix · Stratified Benchmark Suite · Challenge Case Red Team · Problem Distribution Profile · Assumption Register · Regularization Path Review · Out-of-Distribution Monitor · No-Universal-Winner Claim Review · Method Card or Model Card · Algorithm Portfolio Router · Benchmark Refresh Audit
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Scores the candidate method head-to-head against a deliberately assembled ladder of reference points — trivial, incumbent, simple-but-strong, robust, domain-specific, and human-assisted — under identical conditions, so an apparent win has to survive comparison with what it claims to beat, making its operative form a bounded evaluation of existing evidence or work that produces a finding or disposition.
Independent corroboration: The frozen evidence defines Baseline Comparison Table as 'Scores the candidate method head-to-head against a deliberately assembled ladder of reference points — trivial, incumbent, simple-but-strong, robust, domain-specific, and human-assisted — under identical conditions, so an apparent win has to survive comparison with what it claims to beat', so its operative form is Assessment, Review & Assurance.
Nearest alternative: Experiment, Test & Rehearsal — It evaluates candidate performance against a fixed reference ladder, while controlled runs may generate the scores.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Data Science & Analytics
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Specialized
Rationale: Machine-learning evaluation compares a candidate against trivial, incumbent, simple, robust, and domain-specific baselines under identical conditions.
Related originating lineages:
- Computer Science & Software Engineering — Algorithm benchmarking supplies shared tasks and computational baselines.
- Statistics & Experimental Design — Fair comparison requires controlled conditions and prespecified metrics.
Review resolution: Data science is the agreed primary through machine-learning baseline evaluation. Computer science and controlled experimental comparison materially shape it; the page's deliberately complete ladder from trivial through human-assisted baselines is an Encyclopedia synthesis.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
References¶
[1] Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. "Do ImageNet Classifiers Generalize to ImageNet?". In Proceedings of the 36th International Conference on Machine Learning, PMLR 97, 5389–5400 (2019). Finds substantial accuracy drops when strong benchmark classifiers are evaluated on newly collected data. registry ↩