Simpson's Paradox Check¶
Diagnostic check — instantiates Aggregation Bias Detection and Correction
Tests whether an aggregate relationship reverses or materially changes once a confounder or composition variable is conditioned on — the fingerprint of a Simpson reversal.
Simpson's Paradox Check is a targeted diagnostic, not a general breakdown: it asks one sharp question of an aggregate association — does its direction survive conditioning on a plausible confounder? Where a stratified protocol scans every slice for any divergence, this check hunts specifically for the sign flip — the case where an overall relationship points one way and every subgroup points the other. Its second defining move is that it refuses to answer arithmetically: when a reversal appears, a causal argument, not the numbers alone, decides which level to believe. The check therefore carries its own correction rule — retain the pooled reading, adopt the conditioned one, or refuse — rather than assuming the disaggregated number always wins.
Example¶
A company reviews last year's promotions and finds women were promoted at roughly 18% against men at 25%. Read at the top line, it looks like a bias against women. The check conditions on the one composition variable everyone suspects — department. Within nearly every department, women were promoted at an equal or slightly higher rate than men; the aggregate gap came from women being concentrated in departments that promote few people of any gender. The relationship reversed. The check does not stop at the arithmetic: it applies a causal reading — if department is a legitimate pre-existing structure rather than a consequence of the promotion process, the within-department view is the one that speaks to promotion fairness, while the composition itself becomes the real question. The corrected claim: promotion decisions show no adverse gap by gender; the disparity lives upstream in who ends up in which department.[1]
How it works¶
- Assemble candidate markers. List the confounders and composition variables that could plausibly drive the aggregate — the small set worth conditioning on, chosen for causal relevance, not convenience.
- Compute at both levels. Estimate the association in the pool and within each level of each candidate.
- Screen for reversal. Apply the comparison rule tuned to direction and magnitude change — a sign flip or a swing large enough to change the conclusion, not any difference.
- Adjudicate with a causal model. When a reversal is found, decide which level is appropriate using a back-door / collider reading of the variable, then state the corrected claim and its scope.
Tuning parameters¶
- Candidate-marker breadth — how many confounders to test. Wider nets catch more reversals but raise the chance of conditioning on the wrong variable.
- Reversal threshold — sign-flip only vs. any material magnitude change. Stricter thresholds report fewer, cleaner paradoxes.
- Privileged level — which level the correction defaults to, and the causal argument required to override it. Never "always disaggregate."
- Conditioning depth — one variable at a time vs. joint conditioning on several. Joint conditioning is more faithful but thins cells fast.
When it helps, and when it misleads¶
Its strength is exposing composition-driven artifacts — the apparent effect that is really an accident of who is in which group — before they harden into a false causal story. Its failure mode is the mirror image of the fallacy it names: assuming the conditioned number is always the truer one. Sometimes the pooled estimate is correct and the disaggregated one is the artifact — most sharply when the "confounder" is actually a mediator or a collider on the causal path, so conditioning on it manufactures a spurious reversal rather than removing a real one. The classic misuse is exactly this: controlling for a downstream variable to conjure the reversal a stakeholder wants. The guarding discipline is that a causal diagram, not the arithmetic of the flip, decides which conditioning is valid.
How it implements the components¶
confounder_and_composition_marker_set— the curated candidates the check conditions on to explain an aggregate/subgroup divergence.aggregate_subgroup_comparison_rule— here specialized to a reversal test: does the sign or a decision-changing magnitude survive conditioning?interpretation_and_correction_decision_rule— the causal adjudication that picks the trustworthy level and restates the claim.
It does not build or floor the strata it reads — that is Stratified Analysis Protocol — and it does not reweight to a target population, which is Poststratification or Reweighting.
Related¶
- Instantiates: Aggregation Bias Detection and Correction — the reversal diagnostic within the correction loop.
- Consumes: Stratified Analysis Protocol supplies the conditioned breakdowns this check inspects for reversal.
- Sibling mechanisms: Stratified Analysis Protocol · Ecological Fallacy Guardrail · Poststratification or Reweighting · Multilevel Modeling Review · Sensitivity Analysis by Group · Subgroup Dashboard with Warning Flags · Representativeness and Nonresponse Review
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Simpson's Paradox Check operates as an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution because it tests whether an aggregate relationship reverses or materially changes once a confounder or composition variable is conditioned on — the fingerprint of a Simpson reversal.
Independent corroboration: The frozen evidence defines Simpson's Paradox Check as 'Tests whether an aggregate relationship reverses or materially changes once a confounder or composition variable is conditioned on — the fingerprint of a Simpson reversal', so its operative form is Analysis, Modeling & Optimization.
Nearest alternative: Assessment, Review & Assurance — Simpson's Paradox Check includes features of a bounded evaluation of existing evidence or work that produces a finding or disposition, but its defining operation is an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Testing whether an aggregate association reverses after conditioning is the canonical statistical Simpson's paradox diagnostic.
Related originating lineages:
- Data Science & Analytics — Segmented analysis detects composition-driven reversals in operational datasets.
- Economics & Finance — Composition effects can reverse market and policy relationships across groups.
- Mathematics — Mathematical modeling, proof, and abstract-structure practice supplies a parallel or contributing lineage for the mechanism's defining operation: tests whether an aggregate relationship reverses or materially changes once a confounder or composition variable is conditioned on — the fingerprint of a Simpson reversal.
- Medicine & Healthcare — Confounding and risk mix frequently create clinically consequential aggregate reversals.
Review resolution: The blind reviewers agree that statistics_experimental_design is the primary origin and differ only on alternate origin disagreement, domain reach disagreement. I preserve every independently explained alternate from both records rather than imposing a numeric cap. I retain single_lineage because the combined evidence shows one traceable formative lineage. The broader reach of multi_domain records portability separately from historical provenance; encyclopedia_synthesis=false preserves the affirmative synthesis judgment where either reviewer identified one.
Review outcome: Reconciled after independent review; high confidence.
References¶
[1] Simpson's paradox — an association that reverses when a third variable is conditioned on. The canonical real case is the 1973 UC Berkeley graduate-admissions review (Bickel, Hammel & O'Connell, 1975), where an apparent overall bias against women admitting dissolved once applications were examined within department, because women applied disproportionately to more selective departments. withdrawn registry ↩