Prior Sensitivity Analysis¶
Robustness check — instantiates Bayesian Belief Updating
Compares posterior conclusions under several plausible priors to see whether decisions are dominated by starting assumptions.
Prior Sensitivity Analysis re-runs the same update under a deliberately chosen spread of plausible priors and asks a single question: does the conclusion change? If the posterior verdict is the same whether you start skeptical, neutral, or enthusiastic, the evidence is carrying the result; if it flips, the "finding" is really an artifact of the starting assumption. Its defining idea, false of its siblings, is that it produces no belief and no action of its own — it audits the dependence of a conclusion on its prior, and it records where each prior came from so an assumption can be contested on its provenance, not merely on its value.
Example¶
A team pools twelve studies of a tutoring program's effect on standardized test scores using a Bayesian random-effects meta-analysis. The pooled effect turns out to depend heavily on the prior for between-study heterogeneity — how much genuine variation is assumed between the studies. They re-run the identical model under three priors: a vague prior, a weakly-informative half-normal, and a skeptical prior that expects little true variation and small effects.
Under two of the priors the pooled effect is clearly positive; under the skeptical heterogeneity prior the credible interval crosses zero. The prior-source record logs where each prior came from — a standard Cochrane default, a subject-matter expert, and a methodological critic's stance — so reviewers can see that the most enthusiastic prior originated with a program advocate. The reported conclusion becomes explicitly conditional: positive, unless one assumes large unexplained heterogeneity between studies.
How it works¶
- Enumerate a defensible prior set spanning skeptical to enthusiastic — and, in hierarchical models, the variance and heterogeneity priors that quietly dominate the result.
- Re-run the identical update under each, holding evidence and likelihood fixed.
- Compare the posteriors, and the decisions they imply. Flag whether the action-relevant conclusion is stable or flips.
- Record each prior's source and rationale so a contested assumption is traceable and challengeable.
Tuning parameters¶
- Prior spread — how far apart the alternative priors are; too narrow hides fragility, too wide tests straw men.
- Hierarchical focus — whether the variance/heterogeneity priors, often the real driver, are included.
- Stability metric — posterior shift vs. decision flip; the latter is what matters for action.
- Provenance depth — how much source detail is recorded per prior.
- Reporting mode — a single robustness verdict vs. the full grid of posteriors.
When it helps, and when it misleads¶
Its strength is that it is the standard defense against prior lock-in and against laundering a preferred assumption as a neutral input — robust Bayesian analysis[n1] in miniature. Its failure mode is that the chosen prior set can itself be gamed by omitting the one prior that would flip the result, and a decision that survives every prior can still be wrong if the likelihood is misspecified — robustness to the prior is not robustness to the model. The classic misuse is running a token sensitivity check over near-identical priors to manufacture a robustness claim. The guarding discipline is to pre-register the prior set, include the genuinely adversarial prior, and record provenance so the set can be audited by someone who did not choose it.
How it implements the components¶
prior_sensitivity_check— it is the re-run under alternative priors, reporting whether the conclusion is evidence-driven or prior-driven.prior_source_record— it logs each prior's origin and rationale so a contested assumption is traceable and can be challenged on its source.
It runs no forward inference and issues no number of its own: it does not apply the update_rule or build the likelihood_model (that engine is Likelihood-Ratio Reasoning), produce the posterior_estimate it stress-tests (that comes from Bayesian Diagnosis or Sequential Forecast Update), or set a decision_threshold (Adaptive Decision Threshold). It is distinct from calibration_check in Posterior Risk Estimation: calibration asks whether past posteriors matched realized outcomes, while this asks whether today's conclusion survives a change of prior.
Related¶
- Instantiates: Bayesian Belief Updating — the robustness check that keeps priors honest and reviewable.
- Consumes: the update it stress-tests, from Bayesian Diagnosis or Sequential Forecast Update.
- Sibling mechanisms: Bayesian Diagnosis · Likelihood-Ratio Reasoning · Posterior Risk Estimation · Adaptive Decision Threshold · Sequential Forecast Update · Bayesian Model Update
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Prior Sensitivity Analysis operates as an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution because it compares posterior conclusions under several plausible priors to see whether decisions are dominated by starting assumptions.
Independent corroboration: The frozen evidence defines Prior Sensitivity Analysis as 'Compares posterior conclusions under several plausible priors to see whether decisions are dominated by starting assumptions', so its operative form is Analysis, Modeling & Optimization.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Prior Sensitivity Analysis is most plausibly rooted in the statistics_experimental_design tradition because its characteristic form depends on probability, calibrated inference, experimental design, and uncertainty analysis. The assignment tracks that formative lineage, not the many settings in which the mechanism can now be applied.
Related originating lineages:
- Mathematics — The mathematics tradition materially shaped Prior Sensitivity Analysis through its own practice of formal definition, proof, mapping, and quantitative structure.
Review resolution: Both blind reviewers agree that statistics experimental design is the primary origin. Explicit reconciliation resolves alternate origin disagreement, domain reach disagreement. Formative alternate lineages are retained as mathematics; later breadth of use is recorded separately as domain_reach=multi_domain, while origin_mode=cross_disciplinary_synthesis describes the relationship among origin lineages.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Robust Bayesian analysis — James Berger's program of reporting the range of posteriors that arise from a class of reasonable priors, rather than a single prior's answer, so conclusions are stated with their assumption-dependence attached. ↩