Skip to content

Expectation Calibration Review

A calibration review — instantiates Self-Fulfilling Prophecy Interruption

Compares what was predicted against what actually happened while controlling for the treatment the prediction itself caused, separating warranted forecasts from prophecies.

Not every expectation that comes true is a prophecy, and not every accurate forecast should be erased — some risks are real. Expectation Calibration Review is the sibling that guards the whole intervention against naïve positivity by asking the hard question the others skip: was this expectation actually accurate, net of its own effects? Its defining move is to back-test predictions against outcomes while controlling for the confounds the prediction created — unequal opportunity, differential support, intervention exposure — so it can sort expectations into two piles: those that predicted an independent reality, and those that manufactured the reality they "predicted." It interrupts nothing; it audits accuracy, which is what tells the rest of the system where interruption is even warranted.

Example

A venture firm routinely flags certain seed companies "unlikely to reach a Series A." Two years on, most flagged companies indeed failed to raise — apparent vindication of the partners' judgment. A calibration review declines to take the hit rate at face value. It checks what else differed for flagged companies: did they get fewer partner introductions, less follow-on capital, worse terms, thinner attention? Controlling for that withdrawn support, the raw accuracy overstates the forecast's quality — a share of the "failures" were partly produced by the firm's own reduced backing, while a separate share of flags were genuinely well-founded (a dying market no amount of support would save). The review returns not a verdict on any company but a decomposition: which predictions were warranted, and which were self-confirming.

How it works

Its distinguishing discipline is comparison-with-controls, done after the fact:

  • Pair each prior expectation with its later outcome — the raw calibration.
  • Enumerate the confounds — opportunity, support, intervention exposure, and any behavior-channel effect that could have bent the outcome.
  • Control for them — matched comparison or explicit adjustment, so the residual accuracy reflects the prediction, not the treatment it triggered.
  • Sort the result — warranted prediction (keep the signal) versus manufactured confirmation (target for interruption).

Tuning parameters

  • Confounds controlled — which channels are adjusted for; omit the real one and a prophecy passes as accuracy.
  • Control method — matched comparison, statistical adjustment, or a held-out counterfactual; stronger methods need more data.
  • Confound threshold — how much treatment difference counts as "confounded" rather than incidental.
  • Signal preservation — how aggressively to keep predictive information that survives the check, rather than discarding all predictions as suspect.

When it helps, and when it misleads

Its strength is that it keeps the archetype evidence-based in both directions: it stops a true risk from being waved away as "just a prophecy," and stops a self-manufactured outcome from being banked as proof the expectation was right. It is the corrective to base-rate neglect[1] applied to the system's own forecasts.

It misleads when it is weaponized in either direction — used to dismiss every uncomfortable prediction as prophecy (denial of real risk), or run so timidly that no confound is ever "large enough" to matter. It fails outright when there is no clean counterfactual to isolate the treatment effect. The discipline is to specify the confounds and the control method before looking at the hit rate, and to preserve the predictive signal that genuinely survives.

How it implements the components

  • base_rate_and_confound_check — its core: it checks prior evidence, base rates, omitted variables, and unequal opportunity to separate warranted prediction from prophecy.
  • reinforcing_outcome — it interrogates the confirming outcome specifically, asking how much of it was produced by expectation-driven treatment rather than by the target.

It does not build the loop map (expectation_map, Expectation Audit), interrupt any channel (e.g. channel_interruption_choice, Blind Review), or monitor outcomes going forward (outcome_monitor, Outcome Monitoring Review).

Notes

It is easily confused with two siblings and is distinct from both. Prediction Impact Audit asks whether a score changes treatment; this review asks whether the expectation was accurate once that change is removed. Outcome Monitoring Review tracks outcomes forward after an intervention; this looks backward at predictive accuracy. Skipping it is how a prophecy-interruption program tips into denying genuine risk.

References

[1] Base-rate neglect — judging a case by its salient features while ignoring the underlying prior probability. Calibration is the standard corrective: it forces prediction and outcome back onto the same footing, here with the added step of stripping out the treatment the prediction itself caused. ↩