Avoided-Loss Counterfactual Review¶
Retrospective review — instantiates Reflexive Forecast Impact Governance
Judges a forecast that appears to have "failed" by estimating the loss it prevented, so a warning that averts its own prediction is credited as a success rather than a false alarm.
When a forecast provokes the very action that prevents what it forecast, the quiet outcome makes the forecast look wrong. Avoided-Loss Counterfactual Review is the retrospective step that reconstructs what would have happened had no one acted on the forecast — the counterfactual baseline — and credits the gap between that baseline and the observed outcome as avoided loss. Its defining move is that it scores the forecast against the world it changed, not the world that occurred: a self-defeating warning is judged on the harm it displaced. That single reframing is what separates "the forecast was wrong" from "belief in the forecast made it not come true."
Example¶
Consider the Year 2000 problem. Through the late 1990s, forecasts warned that date-handling bugs would fail systems at the century rollover, and organizations spent heavily to remediate. When January 2000 arrived with few visible failures, a narrative formed that the threat had been overblown. An Avoided-Loss Counterfactual Review answers exactly that narrative. Instead of scoring the forecast against the quiet rollover, it reconstructs the no-remediation baseline from the evidence actually available — the defects found and fixed during remediation, the failure behavior of systems that were tested but not yet patched, and problems in the pockets that skipped remediation entirely. The output is a baseline: absent action, roughly some identifiable share of date-dependent systems would have failed, of which a fraction were load-bearing. The verdict is not "nothing happened" but "the forecast drove the action that made nothing happen" — and that case is banked so the next alarmist-looking warning is read against the right baseline rather than dismissed.
How it works¶
- Reconstruct the baseline. Estimate the outcome under no reaction to the forecast — from unacting subgroups, natural experiments, comparable non-acting cases, or the pre-forecast trend.
- Measure the gap. Observed outcome minus baseline is the avoided loss. If the outcome is worse than baseline, the same machinery surfaces an amplified loss — the harmful reflexive case.
- Attribute. Apportion the gap between the forecast's causal contribution and co-occurring causes, so the credit isn't overstated.
- Bank the verdict. Record the avoided- or amplified-loss finding, with its baseline assumptions, into forecast-impact memory as a calibrated prior for similar future forecasts.
Tuning parameters¶
- Baseline construction method — natural experiment (unacting subgroups) versus a modeled counterfactual versus a historical-trend extrapolation. Cleaner causal identification traded against what evidence is actually available.
- Attribution strictness — how much of the avoided loss is credited to the forecast versus other causes. Loose crediting flatters the forecast and invites the skeptic's rebuttal.
- Loss valuation — the units the averted harm is scored in (downtime, dollars, lives), and whether hard-to-price harms are carried qualitatively rather than forced onto one scale.
- Skeptic's discount — how much the estimate is haircut so it survives the "it was never going to happen anyway" critique.
- Review trigger — every forecast, or only those whose outcome undershot the prediction (the candidate false alarms).
When it helps, and when it misleads¶
Its strength is that it defends warnings from being punished for working, and it pulls apart four cases the raw outcome conflates: the forecast was wrong, was right-but-ignored, was self-defeating-and-beneficial (avoided loss), or self-defeating-and-harmful. It is what lets an organization keep funding prevention whose success is, by construction, invisible.
Its central difficulty is that the counterfactual is unobservable, so the estimate is contestable in both directions — inflated to justify the spend, or denied outright by a skeptic. This is the prevention paradox[n1]: the better the prevention, the less visible the averted harm, and the easier it is to call the whole effort a false alarm. The classic misuse is running it backwards — choosing the baseline that makes a past decision look justified. The discipline that guards against it is to fix the baseline method and its assumptions before seeing whether they flatter you, to prefer natural-experiment identification over free modeling, and to carry the skeptic's discount in the open.
How it implements the components¶
Avoided-Loss Counterfactual Review realizes the counterfactual-evaluation side of the archetype — only the components a retrospective review can produce:
counterfactual_success_baseline— its core product: the reconstructed no-reaction outcome the forecast is scored against.forecast_impact_memory— it writes each avoided- or amplified-loss verdict, with its assumptions, into the memory that later reviews and cadences draw on.
It does not measure the forecast's live behavioral footprint — Forecast Impact Audit does that — nor communicate the result to the public, which the Public False-Alarm Explainer handles; and it consumes, rather than produces, the observed-outcome signal owned by the Post-Release Behavior Dashboard.
Related¶
- Instantiates: Reflexive Forecast Impact Governance — it supplies the "did the forecast succeed by defeating itself?" verdict the appraisal needs.
- Consumes: Forecast Release Decision Log for the original claim and release conditions; the Post-Release Behavior Dashboard for the observed outcome.
- Sibling mechanisms: Forecast Impact Audit · Public False-Alarm Explainer · Post-Release Behavior Dashboard · Forecast Release Decision Log · Strategic Gaming Stress Test
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Judges a forecast that appears to have 'failed' by estimating the loss it prevented, so a warning that averts its own prediction is credited as a success rather than a false alarm, making its operative form a bounded evaluation of existing evidence or work that produces a finding or disposition.
Independent corroboration: The frozen evidence defines Avoided-Loss Counterfactual Review as 'Judges a forecast that appears to have 'failed' by estimating the loss it prevented, so a warning that averts its own prediction is credited as a success rather than a false alarm', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Causal inference supplies the defining operation: estimate the unobserved no-intervention outcome and compare it with the observed outcome to identify causal impact.
Related originating lineages:
- Economics & Finance — Avoided-cost and loss valuation translate the causal difference into economic benefit.
- Futurism & Strategic Foresight — Forecast evaluation supplies the reflexive case in which a warning changes the future it predicted.
- Medicine & Healthcare — Prevention evaluation provides canonical cases where successful intervention makes averted harm invisible.
- Public Administration & Policy — Policy evaluation institutionalizes counterfactual appraisal of preventive programs and warnings.
Review resolution: World Bank impact-evaluation guidance defines the counterfactual as what participants would have experienced without the intervention and treats its estimation as the central causal challenge. That method is the review's core, making statistics and experimental design primary; economics, prevention, foresight, and policy materially shape its use, while the forecast-reflexivity memory is synthesized.
Attribution caveat: Economics values avoided loss and policy disciplines apply it, but construction and attribution of the counterfactual baseline are statistical identification problems.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
- World Bank — Impact Evaluation in Practice
- World Bank — Causal Inference and Counterfactual Analysis for Policy Evaluation
Notes¶
The review's output is only as defensible as its pre-registered baseline. Choosing the baseline method after the outcome is known is the single move that turns it from evidence into rationalization — the same hindsight the Forecast Release Decision Log exists to freeze. Decide how you will build the counterfactual before you look at whether it helps your case.
[n1] The observation that a preventive measure which succeeds at the population level brings little visible benefit to identifiable individuals, so successful prevention looks like effort spent on a non-problem — Geoffrey Rose's prevention paradox. It is why avoided-loss cases are so easily re-narrated as false alarms. ↩