Skip to content

Prediction Impact Audit

A model-and-policy audit — instantiates Self-Fulfilling Prophecy Interruption

Checks whether a risk score, forecast, or warning label is changing how a system treats its subjects in ways that help bring about the very outcome it predicts.

When the expectation is a score — a risk rating, forecast, ranking, or warning label produced by a model or policy — the danger is that the prediction quietly becomes an intervention. Prediction Impact Audit is the sibling built for that case. Its defining move is to trace the causal chain from a scored prediction to differential treatment to the predicted outcome, and to test how much of the model's apparent accuracy is confirmation it manufactured. It is set apart from Expectation Audit — which maps human expectations broadly — by two things: it targets automated or scored predictions in triage, compliance, safety, credit, and reputation systems, and it carries the duty to escalate when the loop is producing material harm rather than merely noting it.

Example

A hospital deploys a 30-day-readmission risk score. By policy, high-risk patients get more cautious discharge planning — but in practice the flag also travels into the chart, where rushed staff read it as "revolving-door" and give those patients shorter attention and less thorough discharge teaching. Six months on, high-risk patients readmit more often, and the score looks well-calibrated. The prediction impact audit traces score → treatment change → outcome, isolates the discharge-teaching gap as a channel the score itself opened, and finds that a portion of the "accurate" readmissions was partly manufactured by degraded care. Because this is patient harm, the audit does not file a coaching note — it routes the finding to a formal safety-and-equity review with the authority to change the policy.

How it works

Its distinguishing discipline is auditing a model or policy output for its downstream causal footprint:

  • Trace the score to treatment — identify the specific behavior the prediction changes (attention, scrutiny, resourcing, thoroughness).
  • Test manufactured confirmation — estimate how much of the confirming outcome was produced by that treatment change rather than by independent risk, ideally against a genuine counterfactual (a holdout or matched cohort).
  • Consult the affected subjects — check with those scored and downstream users for treatment they experience as over-surveillance or neglect.
  • Escalate material harm — when the loop produces unfairness, safety risk, or discrimination, route it to formal review rather than absorbing it.

Tuning parameters

  • Which predictions are audited — the scores, forecasts, or labels whose treatment impact is examined.
  • Causal-isolation method — holdout groups, matched cohorts, or natural experiments to separate the score's effect on treatment from real base risk; stronger isolation needs more design.
  • Harm-escalation threshold — what level of demonstrated harm triggers formal review, rights protection, or safety escalation.
  • Stakeholder scope — which affected subjects and downstream users are consulted, and how their reports are weighed.

When it helps, and when it misleads

Its strength is that it catches the specific, dangerous case where a prediction silently becomes a cause — and, unlike a pure diagnostic, it is armed with an escalation path for when the loop is harming people. It is the applied guard against a performative prediction,[1] one that helps bring about the outcome it forecasts.

It misleads when it assumes every accurate score is a prophecy (some predicted risk is genuinely independent) or when it cannot separate the score's treatment effect from real base rates. The classic misuse is running the audit to bless a model — assembling it to certify a system already in production rather than to test it. The discipline is to seek a genuine counterfactual before concluding, keep the base-rate question open, and actually escalate demonstrated harm instead of documenting it into a drawer.

How it implements the components

  • behavior_channel — it identifies the specific path by which the score changes treatment of its subjects.
  • reinforcing_outcome — it tests whether the confirming outcome was partly produced by that score-driven treatment.
  • harm_escalation_path — it routes loops that create material harm to formal review, rights protection, or safety escalation.
  • stakeholder_impact_check — it consults the scored subjects and downstream users about how the prediction-driven treatment is experienced.

It does not fix the loop — standardizing treatment (neutral_treatment_protocol, Neutral Treatment Protocol) and redistributing opportunity (counterexpectation_intervention, Balanced Opportunity Assignment) are interruption siblings — nor does it back-test the score's accuracy net of confounds (base_rate_and_confound_check, Expectation Calibration Review).

Notes

Distinguish it from two cousins. Expectation Audit maps human expectations across a whole process; this audits a scored prediction and uniquely owns the harm-escalation duty. Expectation Calibration Review asks whether the score is accurate net of its effects; this asks whether the score changes treatment to produce the outcome. In algorithmic systems the two are usually run together.

References

[1] Performative prediction — the named phenomenon in which the act of issuing a prediction influences the outcome it forecasts, so the prediction becomes partly self-fulfilling. It is the exact failure this audit is built to detect in risk scores, triage flags, and reputation rankings. ↩