Statistical Paradoxes & Model Reliability¶

← Back to Domain-Specific Families

Abstractions about statistical inference failing through imbalance, leakage, ambiguity, bias, aggregation, underfitting, or misleading evidence. They include classic paradoxes, Bayes factors, regression discontinuity, fairness criteria, confidence errors, small-study effects, and attacks that infer hidden membership.

20 abstractions in this family — domain-specific abstractions that sit near one another in structural-signature space (k-means over structural-signature embeddings). Each is shown with its short description.

Anscombe's Quartet — Four hand-built bivariate datasets that share nearly identical low-order summary statistics yet have radically different scatter geometries, demonstrating that summaries are lossy projections and that one must visualize before granting any parametric summary authority over the data.
Bayes Factor — The ratio of the marginal likelihoods of the data under two specific models, isolating the data's weight of evidence between them and updating prior model odds to posterior odds multiplicatively — with an automatic Occam penalty on flexible models.
Benford's Law — Score a dataset's honesty by checking whether its leading digits follow the fixed logarithmic curve log₁₀((d+1)/d) — about 30% start with 1, only 5% with 9 — that scale-spanning multiplicative data must obey.
Class Imbalance — Diagnose why a 99%-accurate classifier can be useless: when one class vastly outnumbers the class of interest, additive loss aggregation lets majority examples dominate the gradient, so the model learns to ignore the rare minority.
Conservatism Bias — Capture the tendency of people to revise a probability judgment less than Bayes' rule prescribes when evidence arrives — the reported posterior landing short of the correct one, anchored too close to the prior — measured as a signed gap against an explicit Bayesian benchmark.
Ecological Inference Problem — Recover individual-level joint distributions from group-level marginal totals, a many-to-one inverse problem where the data alone only pin the answer to the Duncan-Davis bounds and any tighter estimate rests on an explicit, contestable identifying assumption.
Equalized odds — A classifier fairness criterion requiring protected groups to have equal true-positive and false-positive rates conditional on the actual outcome.
Funnel Plot Asymmetry — Plot each study's effect against its precision and read a departure from the symmetric inverted-funnel expected under unbiased sampling — a gap where small null studies should be — as the visual fingerprint of a publication filter, licensing scrutiny against a fixed set of causes rather than a verdict.
Gambler's Fallacy — Catch the error of believing a run of one outcome makes the opposite 'due' on the next trial — imposing the law of large numbers' aggregate balance as a within-trial obligation — by first screening whether the trial process is independent.
Imputation Leakage — The model-evaluation failure in which a missing-value repair step is fit across the train/test boundary, so its parameters encode facts about the held-out rows — inflating performance that survives into the test metric, because imputation, mentally filed as data cleaning, is really a model.
Jeffreys-Lindley Paradox — The result that a frequentist significance test and a Bayesian posterior-odds comparison of the same data against the same point null can reach opposite verdicts, with the disagreement growing without bound as sample size increases — because the two answer different questions.
Label Ambiguity — Diagnose a headline accuracy figure as a blend of two measurements — model capability in the class interior where annotators agree, and mere adjudication agreement in the boundary zone where reasonable experts split — by stratifying metrics on the inter-annotator agreement rate.
Membership Inference Attack — Determine whether a specific record was in a model's training set by reading the systematic behavioural gap — lower loss, higher confidence — between the samples it saw and statistically matched samples it did not.
Monotone Likelihood Ratio Property — Order a parametric family so every higher-parameter to lower-parameter likelihood ratio is nondecreasing in one statistic, making larger statistics monotonically stronger evidence for the higher parameter.
Regression Discontinuity Design — Recover a causal effect from a threshold rule by comparing units just above and just below a sharp cutoff on a continuous running variable, where they are comparable in expectation, so any jump in the outcome at exactly the cutoff is attributable to the treatment rather than to selection.
Reliability Paradox — Explain why tasks with robust group-level effects (Stroop, IAT) can be useless for ranking individuals: the design minimized within-subjects error for group power without guaranteeing the between-subjects variance that reliability, true-score over total variance, requires.
Small-Study Effects — The meta-analytic pattern in which smaller studies report systematically larger effects than larger ones, producing funnel-plot asymmetry that inflates the pooled estimate — a shared symptom of several biases, not a diagnosis of any one cause.
Type M Error — Quantify how much a significant effect's reported magnitude is exaggerated by the significance filter under low power, via the exaggeration ratio — the expected significant estimate divided by the true effect — computable from the design before any data exist.
Type S Error — Quantify the risk that a statistically significant estimate points the wrong way by computing, before data collection, the probability that a two-sided significance filter is cleared from the opposite tail when the true effect is near zero relative to noise.
Underfitting — The failure mode where a model's hypothesis class is too restrictive to capture the structure genuinely present in the data — high bias, with training and test error both elevated and close together — curable only by a richer functional form, not by more data or regularization.