Calibration Adjustment Rule¶
Control rule — instantiates Heuristic Calibration and Confidence Judgment
Converts a measured miscalibration into a standing transform that reshapes every future confidence claim before it is acted on.
A Calibration Adjustment Rule is the standing correction you bolt onto a heuristic once you have measured how wrong its confidence tends to be. It does not re-examine each case, route it, or plot it — it takes the raw confidence the heuristic emits and rewrites the number by a fixed, evidence-derived transform: shrink an inflated 90% down to the 72% the track record actually supports, widen an interval that has been too narrow, or shift a whole band toward the base rate. Its defining move is that it operates between reviews, silently and every time, so that the confidence a decision-maker sees has already been corrected for the heuristic's known bias. Where a bucket review finds the error and a recalibration loop decides to re-fit, the adjustment rule is the compiled output of those findings — a small function that changes the claim itself.
Example¶
A consumer-lending team runs a fast underwriting heuristic that assigns each applicant a "confidence of repayment" grade from a handful of cues — income stability, existing obligations, thin-file flags. A back-test against two years of resolved loans shows the heuristic is systematically overconfident in its top band: applicants it grades at "95% likely to repay" actually repay about 84% of the time, and the gap widens for self-employed applicants. Rather than retrain the whole model or convene a meeting per file, the team writes an adjustment rule: subtract a fixed overconfidence margin from every grade above the 90 line, apply a larger margin when the self-employed flag is set, and cap any single-cue-driven grade at 88 until more evidence accrues. From then on the number the loan officer sees is the corrected one. The rule is versioned and dated, so when the next back-test lands the team can see which correction was live for which cohort — and retire it if the market shifts and the overconfidence disappears.
How it works¶
The rule has three moving parts. First, a direction and shape: is the heuristic over- or underconfident, and is the error flat across the range or concentrated at the extremes? A uniform overconfidence gets a constant shrink; an error that grows with claimed confidence gets a slope. Second, a conditioning key: the correction may differ by segment, regime, or cue, so the rule carries a small register of where each variant applies. Third, a transform: the actual arithmetic — a subtraction, a multiplicative shrink toward the base rate, an interval-widening factor, or a hard cap. The transform is fit to the measured error profile, not chosen by taste, and it is applied deterministically so two identical inputs always get the same corrected output. The rule is deliberately dumb at run time; all the intelligence lives in how it was fit and when it is retired.
Tuning parameters¶
- Shrink strength — how hard the rule pulls confidence toward the base rate. Stronger shrinkage kills overconfident errors but flattens genuine discrimination, so a well-calibrated-but-timid heuristic loses useful sharpness.
- Conditioning granularity — one global correction versus per-segment variants. Finer keys fit the real error better but split the evidence thinner, inviting corrections fit to noise.
- Cap placement — where the hard ceiling on confidence sits. A low cap is safe under uncertainty but forfeits the speed benefit of ever acting on a genuinely high-confidence call.
- Refit cadence — how often the transform is re-derived. Frequent refits track drift but chase sampling wobble; rare refits are stable but go stale after a regime change.
- Sunset condition — the evidence threshold at which a correction is retired. Too eager and you thrash; too reluctant and you keep down-weighting a heuristic that has since improved.
When it helps, and when it misleads¶
Its strength is leverage: one small, auditable function repairs a systematic bias across thousands of judgments without slowing any of them or asking anyone to think harder in the moment. It is the cleanest way to make trust track reliability when the miscalibration is stable and directional.
Its failure mode is that a fixed transform assumes the error it was fit to is still the error you have. When the environment shifts, yesterday's shrink becomes today's distortion — you keep subtracting an overconfidence margin the heuristic no longer has, and now you are underconfident and slow. The classic misuse is fitting an aggressive correction to a thin or cherry-picked sample and then trusting the tidy corrected number more than the raw one deserved — false precision laundered through arithmetic. The guarding discipline is to treat the rule as provisional: version it, attach it to the exact error profile that justified it, and bind its life to a sunset condition so it dies when its evidence does. A useful sanity check is regression to the mean — extreme confidence claims should be pulled inward roughly as far as extreme past outcomes regressed.[n1]
How it implements the components¶
calibration_error_profile— the rule is the compiled form of the profile: it encodes the measured over/underconfidence (direction, shape, magnitude) directly as its transform.confidence_claim_format— it operates on and re-emits the standardized confidence claim, so the corrected number arrives in the same format downstream consumers already read.boundary_condition_register— its conditioning key is a register of where each correction variant applies (which segment, regime, or cue), lowering or reshaping confidence at those boundaries.
It does not gather outcomes or reassign ownership through a feedback_collection_loop or calibration_owner — that periodic re-fitting is Post-Outcome Recalibration Review; nor does it route low-confidence cases through an escalation_and_override_gate — that is Low-Confidence Escalation Trigger, which changes who decides rather than changing the number.
Related¶
- Instantiates: Heuristic Calibration and Confidence Judgment — supplies the standing correction that keeps emitted confidence matched to demonstrated reliability.
- Consumes: Confidence Bucket Review or Reliability Diagram or Calibration Curve — the error measurement the transform is fit to.
- Sibling mechanisms: Challenge Case Set · Confidence Bucket Review · Ecological Validity Screen · Expert Disagreement Calibration · Low-Confidence Escalation Trigger · Post-Outcome Recalibration Review · Prediction Journal · Reference Class Comparison · Reliability Diagram or Calibration Curve
Editorial Notes¶
Form Classification¶
Form family: Control, Automation & Runtime
Rationale: The mechanism applies a segment- or regime-specific correction transform to every incoming confidence claim before action, so its operative form is runtime calibration control.
Nearest alternative: Rule, Policy & Commitment — The transform is a standing rule, but repeated execution over live claims is the deployed mechanism.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Statistical calibration turns empirical forecast error into a fitted transform that maps raw confidence to observed frequency or corrected interval width.
Related originating lineages:
- Data Science & Analytics — Production modeling implements the transform on every later score before a decision is made.
- Psychology — Judgment research identifies systematic overconfidence and underconfidence that require correction.
Review resolution: Statistics is the agreed primary lineage because recalibration rules correct systematic prediction or measurement error from observed residuals. Data science and psychology are formative applications, while the underlying adjustment rule remains an established statistical lineage.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Regression to the mean — extreme observed values tend to be followed by less extreme ones because part of the extremity was chance. It is the statistical warrant for shrinking extreme confidence claims toward the base rate: the more a claim relies on a small or noisy signal, the harder it should regress. ↩