Causal Inference Review¶
Review procedure — instantiates Causal Mechanism Mapping
Audits the identification assumptions, comparison groups, confounders, and evidence behind a causal claim before it is accepted.
A Causal Inference Review is a gate, not a build. Someone else has already produced a causal claim — often a study, model, or analysis with a number attached — and this procedure interrogates whether the claim's identification actually holds before anyone acts on it. It walks a skeptical checklist: what is the comparison being made, and is it a credible counterfactual? What confounders could produce the association without the causal link? What assumptions does the chosen method silently require, and are they plausible here? Its defining stance is adversarial and negative — it can and should reject a claim for weak identification, whereas the other mechanisms in this archetype construct pathways or contribution stories. It does not gather new evidence or run the pathway; it pressure-tests the reasoning that connects the evidence already in hand to the causal conclusion.
Example¶
An economics policy unit receives a consultant's report concluding that a recent minimum-wage increase in one state "reduced teen employment by 4%," based on comparing that state's teen-employment change against the national average over the same year. A reviewer runs a causal inference review before the finding reaches a legislative briefing. She starts with the counterfactual: is "the national average" a credible stand-in for what this state would have done absent the policy? The state's economy is oil-linked, and oil prices fell that year — a confounder that depresses local employment regardless of the wage floor, and one the national comparison does not remove.
She then probes the identification assumptions of the difference-in-differences the report leans on: did the two groups' employment trends move in parallel before the policy? Plotting the pre-period shows they did not — the state was already diverging — which undercuts the method's key assumption. Finally she lists rivals the report ignored: a concurrent change in how the state coded seasonal agricultural workers, and simple regression to the mean after an unusually high prior year. Her review does not produce a new estimate; it returns a verdict — not accepted as causal; identification is not established — with the specific weaknesses named and the comparison and confounders flagged for the next analysis to fix. That gate is what stops a fragile 4% from hardening into policy fact.
How it works¶
- Reconstruct the implicit comparison. Every causal estimate rests on a counterfactual — treated versus what-would-have-happened. The review's first job is to surface that comparison and judge its credibility.
- Stress the identifying assumptions. Each method (difference-in-differences, matching, instrumental variables, regression discontinuity) buys causality with assumptions; the review names them and asks whether they hold in this setting, often by checking testable implications like pre-trends.
- Hunt confounders and rivals the claim omits. It actively adds back the common causes, selection effects, and measurement changes the original analysis may have missed, and asks whether any could produce the result on its own.
- Return a verdict, not an estimate. The output is accept / accept-with-caveats / reject-for-weak-identification, with the specific fixes the claim would need — it critiques rather than re-computes.
Tuning parameters¶
- Skepticism threshold — how strong identification must be before "accept." A high bar avoids acting on fragile claims but can stall decisions that must be made under irreducible uncertainty.
- Assumption depth — how many of a method's identifying assumptions are actually tested versus taken on faith. Deeper audits catch more but require access to the underlying data.
- Rival-generation effort — how hard the reviewer works to invent omitted confounders. More effort finds more holes but risks nihilism where every claim is deemed unprovable.
- Verdict granularity — binary accept/reject versus a graded confidence with conditions. Graded verdicts are more useful downstream but blunt the gate's decisiveness.
- Independence — whether the reviewer is separate from the claim's authors. Independence sharpens the critique; embedding speeds it but invites capture.
When it helps, and when it misleads¶
Its strength is catching the fragile causal claim before it drives a decision — the impressive study whose comparison group is wrong, whose parallel-trends assumption fails, or whose headline confounder was never adjusted for. As a standing gate on analyses entering high-stakes decisions, it is the cheapest insurance against acting on a laundered correlation.
Its failure mode is methodological nihilism: because no observational claim has airtight identification, a reviewer optimizing to look rigorous can reject everything, leaving decision-makers with critique but no usable answer — infinite skepticism that never has to be right about anything.[n1] The classic misuse is demanding randomized-trial certainty from a question where randomization is impossible and then treating the resulting "not established" as license to ignore strong-but-imperfect evidence. The guarding discipline is to pair every rejection with what would raise confidence — the comparison to fix, the assumption to test, the data to collect — so the review improves the next analysis instead of merely dismissing this one; and to calibrate the skepticism threshold to the decision's stakes rather than to the reviewer's appetite for looking careful.
How it implements the components¶
Causal Inference Review fills the identification-audit components — the ones a skeptical gate carries:
confounder_check— its core service: systematically adding back the common causes and selection effects the claim may not have removed, and judging whether the association survives them.counterfactual_anchor— it reconstructs and grades the comparison the estimate rests on (control group, pre/post baseline, matched or synthetic comparison), the thing on which identification stands or falls.rival_explanation_set— it generates the alternative explanations the original analysis omitted and asks whether any alone could produce the result.
It neither builds the positive pathway — mechanism_chain, filled by Mechanism Map — nor manipulates anything: intervention_point and causal_test belong to Intervention Test. This review only scrutinizes an existing claim's identification on paper; it recommends a test where one is needed but does not run it.
Related¶
- Instantiates: Causal Mechanism Mapping — supplies the identification-and-rivals guardrail the archetype requires before a causal claim is accepted.
- Sibling mechanisms: Contribution Analysis · Intervention Test · Causal Diagram · Mechanism Map · Causal-Loop Map · Theory of Change Model · Failure Tree Analysis · Process Tracing · Root-Cause Analysis
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Audits the identification assumptions, comparison groups, confounders, and evidence behind a causal claim before it is accepted, making its operative form a bounded evaluation of existing evidence or work that produces a finding or disposition.
Independent corroboration: The frozen evidence defines Causal Inference Review as 'Audits the identification assumptions, comparison groups, confounders, and evidence behind a causal claim before it is accepted', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Convergent development
Present-day reach: Multi-domain
Rationale: Causal methodology established adversarial review of counterfactuals, confounding, comparison groups, and identification assumptions.
Related originating lineages:
- Economics & Finance — Econometrics supplies method-specific assumptions for instruments, discontinuities, panels, and natural experiments.
- Philosophy — Philosophy of causation contributes counterfactual warrant and the limits of causal claims from observational evidence.
Review resolution: Statistics and experimental design is the agreed primary lineage because the review audits counterfactual comparisons, confounding, and identifying assumptions. Econometrics and philosophy independently contribute design-specific assumptions and causal justification, so convergent is more accurate than a single lineage.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
The nearest sibling is Contribution Analysis, and the two are complementary opposites. Both grapple with imperfect causal evidence, but the review is a skeptical gate that can reject a claim for weak identification, while contribution analysis is a constructive method that builds a positive, rival-tested plausibility argument when clean identification was never available. Run the review to decide whether to trust someone's estimate; run contribution analysis when no estimate can be cleanly identified and you must still argue a defensible verdict.
[n1] The failure of treating the impossibility of perfect causal identification as grounds to reject all causal claims — a posture that is always defensible and never useful. The corrective, associated with the modern credibility-revolution literature in econometrics, is to grade identification on a spectrum and pair skepticism with the specific improvement that would raise confidence. ↩