Model Comparison Table¶
Comparison artifact — instantiates Assumption-Light Inference
Lays the same question's answers side by side under strong and assumption-light frames, turning method disagreement into a visible, decidable finding.
When two analysts disagree less about the data than about the assumptions imposed on it, the resolution is not to pick a winner in the abstract but to show the answers next to each other. Model Comparison Table is the artifact that does this: a single grid whose rows are analytic frames — the strong parametric model, one or more assumption-light alternatives — and whose columns are the decision-relevant outputs each frame produces (the estimate, the interval, the direction, the call). Its defining move is that agreement and divergence become the finding. If every frame points the same way, the conclusion is robust and can be acted on despite imperfect assumptions; if the frames split, the divergence is the result — a signed notice that the answer depends on a contested assumption and must be softened, bounded, or investigated rather than reported as settled.
Example¶
A bank is deciding whether a new underwriting rule lifts approval rates without raising defaults. One analyst fits a logistic regression with the standard assumptions; another worries the default counts are sparse and clustered by branch. Instead of arguing, they build a comparison table. The rows: the standard logistic model, a cluster-robust version, a rank-based comparison of default rates, and a resampled stability check. The columns: estimated default-rate change, its interval, and the go/no-go each frame implies.
Three of the four rows agree — defaults are essentially flat and the rule is safe. The fourth, the cluster-robust frame, widens the interval enough to cross zero. The table makes the situation legible in one glance: the point estimate is stable, but its significance hinges entirely on whether branch clustering is real. That is the finding the meeting needed. Rather than adopt or reject the rule on one analyst's model, the bank chooses the assumption-light reading for the decision and commissions a clustering check to resolve the one row that dissents.
How it works¶
- Fix the question, vary the frame. Hold the decision-relevant output constant across rows; each row is the same question answered under a different assumption set (strong model, robust, rank-based, resampled).
- Choose the frames deliberately. The alternative frames are not arbitrary — each is selected to relax a specific assumption the strong model makes, so a divergence localizes to a named assumption.
- Read agreement and divergence. Concordance across frames is evidence of robustness; discordance flags the assumption the answer is hostage to, and points at which frame's assumption to test next.
- Hand off, don't adjudicate. The table surfaces where the frames split; it does not itself declare the winner or write the caveat — it equips the decision and the downstream limit statement.
Tuning parameters¶
- Frame set — how many and which alternative models populate the rows. More frames map the assumption space better but dilute attention and can manufacture spurious disagreement.
- Output columns — point estimate only, or estimate plus interval plus decision. Richer columns reveal whether divergence is in the magnitude or only in the uncertainty, at the cost of a busier grid.
- Concordance rule — how close the frames must agree to be called "robust." A strict rule flags fragility aggressively; a loose one risks blessing a shaky conclusion.
- Ordering — strong-to-light versus light-to-strong. The ordering subtly frames which model is treated as the baseline the others are checked against.
When it helps, and when it misleads¶
Its strength is that it makes assumption dependence visible and decidable in a way prose never does: a decision-maker who cannot follow the statistics can still read whether the columns agree. It is the natural home for a multiverse analysis — reporting the result across the reasonable set of analytic choices rather than cherry-picking one — which is the discipline's answer to selective specification.[1]
Its failure mode is a rigged grid. Because the builder chooses which frames to include, a table can be curated to show consensus by omitting the one frame that would dissent, or to fabricate false controversy by padding it with strawman models nobody would use. It also only compares the frames it contains — a shared blind spot across every row (a confounder, a biased sample) sails through looking robust, because agreement among models that make the same mistake is not robustness. The classic misuse is treating column concordance as proof of correctness rather than of insensitivity to the assumptions varied. The guarding discipline is to pre-commit the frame set to the assumptions actually at issue (from the audit), and to remember the table certifies robustness only against the frames on it.
How it implements the components¶
sensitivity_comparison— the table is the sensitivity comparison: it shows directly whether the conclusion changes as the analytic frame moves from strong-model to assumption-light.assumption_light_method_choice— by exhibiting which frame the answer should hang on when the frames diverge, it drives the selection of the assumption-light method the decision adopts.
It compares and selects among frames but neither generates the resampled figures a row might display nor writes the final caveat. It does NOT implement resampling_or_permutation_plan — that belongs to Bootstrap-Like Checks, which produces the stability row this table can tabulate — nor interpretation_limit, which a summary mechanism attaches downstream.
Related¶
- Instantiates: Assumption-Light Inference — supplies the cross-frame comparison that decides whether a conclusion is robust.
- Consumes: Bootstrap-Like Checks, Robust Statistics, Nonparametric Tests — each can supply a row the table lays alongside the strong model.
- Sibling mechanisms: Assumption Audit Checklist · Diagnostic Plot Review · Median-Based Summaries · Bootstrap-Like Checks · Nonparametric Tests · Permutation Tests · Rank-Based Methods · Robust Statistics
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: The mechanism holds the question fixed while comparing results under deliberately varied assumption sets to infer which conclusions are robust or assumption-dependent.
Nearest alternative: Representation, Specification & Plan — A table presents the results, but the operative value comes from comparative analysis rather than the static layout.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Side-by-side comparison across defensible model specifications follows statistical robustness, specification-curve, and multiverse-analysis practice.
Related originating lineages:
- Data Science & Analytics — Applied analytics contributes practical comparison of predictive models and deployment tradeoffs.
- Philosophy — Philosophy of science contributes comparison of explanatory frames and assumptions.
Review resolution: Both independent reviews agree on primary origin statistics_experimental_design; reconciliation resolves secondary fields (reported_ambiguity, alternate_origin_disagreement, origin_mode_disagreement, encyclopedia_synthesis_disagreement). Alternate origins retained (data_science, philosophy) are the union of reviewer-supported formative lineages with explicit rationales, not a list of later application domains. Present-day breadth is represented separately as domain_reach=multi_domain; origin_mode=cross_disciplinary_synthesis records the historical relationship among lineages. Confidence is conservatively reconciled to medium, and encyclopedia_synthesis=true preserves either reviewer's finding that the encyclopedia generalized the mechanism.
Attribution caveat: The artifact is a synthetic presentation format rather than a uniquely inherited method.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; medium confidence.
References¶
[1] Multiverse analysis (Steegen, Tuerlinckx, Gelman & Vanpaemel, 2016) — reporting a result across the full set of defensible data-processing and modeling choices rather than a single path, so a claim's dependence on arbitrary analytic decisions is exposed instead of hidden. A model comparison table is the compact, decision-facing form of the same idea. withdrawn registry ↩