Sensitivity Analysis by Group¶
Robustness analysis — instantiates Aggregation Bias Detection and Correction
Re-runs the aggregate under alternative groupings, weights, windows, and exclusions to see whether the conclusion survives the choices that produced it.
Sensitivity Analysis by Group treats every discretionary analytic choice — how groups are cut, which weight is used, which time window, which cases are excluded — as a knob, and asks how much the conclusion moves as those knobs turn. Its defining move is to run the aggregate across a whole space of defensible specifications rather than one, and to report the distribution of results. A conclusion is trusted only if it is stable across the choices that could reasonably have been made; a conclusion that hangs on one particular grouping is flagged as fragile. Where other siblings correct a single aggregate, this one stress-tests whether the aggregate deserved to be believed at all.
Example¶
An economic brief claims regional wage growth was about 4% last year. Before it informs policy, the analyst runs a group-sensitivity sweep. The defensible knobs: industry grouping (2-digit vs. 3-digit sectors), the accounting window (calendar vs. fiscal year), exclusions (drop top-coded high earners; drop part-time workers), and the weight (headcount vs. hours worked). Crossing these yields roughly two dozen specifications; the estimates spread from about 1.8% to 4.2%. The headline "4%" turns out to sit near the top of that range — reachable only under one particular grouping and window. The provenance log records every specification and its result, and the decision rule delivers the verdict: the conclusion is specification-dependent, with a modal estimate closer to 3%, and the brief is rewritten to report the range and name the choices the top figure depended on.[n1]
How it works¶
- Enumerate the defensible choices. List the grouping, weighting, windowing, and exclusion knobs, and the reasonable settings for each — pre-specified so the space isn't drawn to flatter a result.
- Run the grid. Compute the aggregate under each combination of settings.
- Record every run. Log each specification and its outcome in the provenance record.
- Read the distribution. Summarize the spread and apply the decision rule: robust (tight cluster), fragile (wide or sign-flipping), or specification-dependent (clustered but pivoting on one named choice).
Tuning parameters¶
- Choices varied — which knobs enter the sweep. Too few misses fragility; too many make almost anything look fragile.
- Specification count — how densely the space is sampled. Denser sweeps are more honest but costlier and harder to summarize.
- Defensibility bar — what qualifies as a legitimate specification. A loose bar dilutes the signal with indefensible runs; a strict, pre-declared bar keeps it meaningful.
- Robustness threshold — how tightly results must cluster to call a conclusion "robust."
- Summary form — a specification curve, a min/max envelope, or the share of specifications supporting the claim.
When it helps, and when it misleads¶
Its strength is catching the fragile or cherry-picked result — the finding that exists only because of one convenient cut — before it is acted on. Its failure mode is that the space itself can be gamed: pack it with weak specifications and a real effect looks unstable, or make it so vast that everything appears fragile. The classic misuse is the sweep run in reverse — the garden of forking paths, where an analyst quietly explores many specifications and reports only the one that supports the desired claim, the opposite of what the mechanism is for. The guarding discipline is to pre-specify the defensible space and report the entire curve, not a hand-picked point on it.
How it implements the components¶
disaggregation_plan— the enumerated grid of alternative grouping and disaggregation specifications is the plan, generalized from one breakdown to many.aggregation_provenance_record— every specification and its result are logged, making the sweep auditable and reproducible.interpretation_and_correction_decision_rule— the robust / fragile / specification-dependent verdict that decides what claim, if any, survives.
It does not name a single canonical subgroup map or enforce cell-size floors — that is Stratified Analysis Protocol — and it does not reweight to one target population, which is Poststratification or Reweighting.
Related¶
- Instantiates: Aggregation Bias Detection and Correction — the robustness stress-test in the correction loop.
- Consumes: Stratified Analysis Protocol supplies the base disaggregations the sweep perturbs.
- Sibling mechanisms: Stratified Analysis Protocol · Simpson's Paradox Check · Ecological Fallacy Guardrail · Poststratification or Reweighting · Multilevel Modeling Review · Subgroup Dashboard with Warning Flags · Representativeness and Nonresponse Review
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Sensitivity Analysis by Group operates as an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution because it re-runs the aggregate under alternative groupings, weights, windows, and exclusions to see whether the conclusion survives the choices that produced it.
Independent corroboration: The frozen evidence defines Sensitivity Analysis by Group as 'Re-runs the aggregate under alternative groupings, weights, windows, and exclusions to see whether the conclusion survives the choices that produced it', so its operative form is Analysis, Modeling & Optimization.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Convergent development
Present-day reach: Multi-domain
Rationale: Re-estimating results under alternative stratifications, weights, windows, and exclusions is statistical subgroup and robustness analysis.
Related originating lineages:
- Data Science & Analytics — Analytical pipelines routinely perturb segmentation and aggregation choices.
- Mathematics — Mathematical modeling, proof, and abstract-structure practice supplies a parallel or contributing lineage for the mechanism's defining operation: re-runs the aggregate under alternative groupings, weights, windows, and exclusions to see whether the conclusion survives the choices that produced it.
- Medicine & Healthcare — Epidemiology institutionalized stratified analyses and sensitivity to case definitions.
- Sociology & Anthropology — Social categories and population definitions materially alter aggregate interpretation.
Review resolution: The blind reviewers agree that statistics_experimental_design is the primary origin and differ only on alternate origin disagreement, origin mode disagreement, encyclopedia synthesis disagreement. I preserve every independently explained alternate from both records rather than imposing a numeric cap. I retain convergent because the combined record shows independent disciplinary development. The broader reach of multi_domain records portability separately from historical provenance, and encyclopedia_synthesis=true preserves the affirmative synthesis judgment where either reviewer identified one.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Multiverse or specification-curve analysis — running an outcome across the full set of defensible analytic choices and reporting the distribution of results rather than a single preferred estimate (Steegen et al.; Simonsohn, Simmons & Nelson). Its motivating hazard is Gelman & Loken's "garden of forking paths," where undeclared analytic flexibility inflates the chance of a spurious finding. ↩