Proxy Drift and Goodhart Audit¶
An audit — instantiates Construct–Proxy–Signal Validity Alignment
Periodically re-checks whether a proxy still tracks its construct once people are optimizing it — catching the moment a measure-turned-target decouples and needs revision.
Validity is not established once and kept forever — a proxy that genuinely tracked its construct at launch can silently stop, especially once people start optimizing the proxy instead of the thing it stands for. Proxy Drift and Goodhart Audit is the mechanism built for that failure. Where the other checks establish validity at a point in time, this one assumes the proxy is under pressure and watches the construct–proxy relationship over time, looking for the gap between them widening. Its defining premise is Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. So the audit re-measures on a cadence against a harder-to-game reference, and it carries a pre-committed revision rule — the point at which observed drift means the proxy must be rebalanced or retired.
Example¶
A customer-support organization uses "tickets closed per day" as a proxy for support effectiveness, and ties it to team dashboards. At launch the proxy tracked the construct well — good support did close tickets. Six months later the audit re-checks it against a harder-to-game reference: repeat-contact rate and a sampled quality-review score. Tickets-closed is up 30%, but repeat contacts are up and quality-review scores are down — agents are closing tickets prematurely and reopening churn to hit the number. The proxy has decoupled from the construct it was meant to represent; it now measures throughput of closures, not effectiveness. The pre-committed rule fires: rebalance the metric (pair it with resolution-stickiness) or retire it as a target.
How it works¶
- Baseline the proxy–construct link against a reference. Anchor the proxy to a harder-to-game measure of the construct while the two are still aligned, so drift has a yardstick.
- Re-measure on a cadence and watch the gap. Track the proxy and the reference together; the signature of trouble is the proxy moving while the construct stays flat or falls — the fingerprint of optimization pressure.
- Fire the revision rule. When drift crosses a pre-set threshold, the rule triggers a defined response — retrain, rotate, rebalance, or retire the proxy — rather than leaving it to negotiation after the fact.
Tuning parameters¶
- Monitoring cadence — how often the check runs; too slow and gaming compounds before detection, too fast and noise triggers false alarms.
- Reference measure — what un-incentivized yardstick the proxy is checked against; the whole audit is blind if the reference is itself gamed.
- Drift threshold — how large a divergence counts as decoupling; a tighter threshold catches problems earlier but cries wolf more.
- Revision response — what firing the rule does — rebalance, rotate the metric, or retire it; pre-committing this is what stops a convenient metric from being defended past its usefulness.
When it helps, and when it misleads¶
Its strength is catching slow validity decay that any one-time validation misses entirely — the construct–proxy link that was fine on day one and hollow by month twelve, precisely because the proxy became a target.[n1]
Its limits: it needs a trustworthy reference the incentives haven't already corrupted (if the anchor is gamed too, the audit is blind), and there is always a detection lag during which the gaming runs. The classic misuse is explaining away observed drift as a data artifact in order to keep a convenient, familiar metric alive. The discipline is to pre-commit the revision rule before the metric is under pressure, and to guard at least one reference measure from ever becoming a target itself.
How it implements the components¶
validity_monitoring_and_revision_rule— its core: the cadence, the drift threshold, and the pre-committed trigger that revises or retires the proxy.confound_and_surrogate_boundary— by tracking the widening gap, it detects when the proxy has become a gamed surrogate diverging from the construct it was meant to represent.
It does not establish validity at the outset (that is the Multi-Trait Multi-Method Matrix, Factor-Structure or Latent-Model Check, and Known-Groups or Contrast-Case Test), nor does it test cross-group equivalence at a single point in time (that is the Measurement Invariance Audit).
Related¶
- Instantiates: Construct–Proxy–Signal Validity Alignment — it supplies the over-time monitoring that keeps a once-valid proxy from silently decaying.
- Sibling mechanisms: Measurement Invariance Audit · Multi-Trait Multi-Method Matrix · Construct Validity Argument · Content-Domain Review Panel · Construct-to-Proxy Traceability Table · Cognitive Interview or Response-Process Probe · Factor-Structure or Latent-Model Check · Known-Groups or Contrast-Case Test · Validity Limitation Memo
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Proxy Drift and Goodhart Audit operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it periodically re-checks whether a proxy still tracks its construct once people are optimizing it — catching the moment a measure-turned-target decouples and needs revision.
Independent corroboration: The frozen evidence defines Proxy Drift and Goodhart Audit as 'Periodically re-checks whether a proxy still tracks its construct once people are optimizing it — catching the moment a measure-turned-target decouples and needs revision', so its operative form is Assessment, Review & Assurance.
Nearest alternative: Monitoring, Sensing & Alerting — Proxy Drift and Goodhart Audit includes features of ongoing observation, sensing, or alerting that detects and surfaces state without itself executing the response, but its defining operation is a bounded evaluation of existing evidence or work that produces a finding or disposition.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Economics & Finance
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Universal
Rationale: The page explicitly operationalizes Goodhart's monetary-policy insight about targeted indicators losing validity.
Related originating lineages:
- Data Science & Analytics — Model and feature-drift monitoring supplied the repeated empirical audit machinery.
- Organizational & Management Science — Performance-measurement systems supplied the institutional setting in which targets reshape behavior.
- Public Administration & Policy — Campbell's law independently developed the corruption of social indicators under administrative stakes.
- Statistics & Experimental Design — Construct validation and repeated calibration supply the audit's measurement method.
Review resolution: Both blind reviewers agree on economics_finance as the primary origin. Explicit reconciliation resolves alternate_origin_disagreement. The merged alternate lineages retain only domains the reviewers identified as materially formative; domain_reach=universal records later applicability separately from origin breadth.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Goodhart's Law — in Strathern's often-quoted formulation, "when a measure becomes a target, it ceases to be a good measure" — and Campbell's Law about the corruption of social indicators under high stakes describe exactly the decay this audit is designed to detect. ↩