Proxy Validity Audit¶

Proxy-validity audit — instantiates Sense-Experience Reduction Protocol

Checks whether a proxy or instrument actually tracks the target property, and logs its calibration and reliability over time.

Proxy Validity Audit interrogates a standing question the mapping mechanisms cannot answer for themselves: does this proxy still deserve to stand in for the thing it measures? Its defining move is scrutiny over time. Where Instrument Readout Mapping builds the rule "readout means value," this audit repeatedly asks whether the rule is still valid — whether the proxy genuinely covaries with the target property and how reliably it does so — and keeps a running record of calibration and drift. It exists because proxies decay: a stand-in that tracked the target when it was chosen can quietly detach, so that a stable, confident readout maps to a value that has silently moved.

Example¶

A support organisation has long used tickets closed per agent per day as its proxy for customer satisfaction — the intuition being that faster resolution means happier customers. A Proxy Validity Audit tests that link rather than assuming it. It pulls the proxy alongside a more direct read of the target (post-contact satisfaction surveys, repeat-contact rates) and asks whether they still move together. They no longer do: closures are up ≈20% while satisfaction has slipped, because agents have learned to close tickets fast by resolving the easy part and letting customers re-open the rest.

The audit's output is a validity finding plus a reliability record: the proxy has detached from the target under pressure, its correlation has decayed, and its reliability across agents is uneven. That verdict is what tells the org to stop trusting the closures number as a satisfaction signal — not because the number is wrong, but because it no longer maps to the thing anyone cared about.

How it works¶

Name the target and the proxy separately. State the property that matters and the observable standing in for it, refusing to let the proxy silently become the target.
Test covariation against a more direct read. Compare the proxy with a criterion closer to the target — a survey, a gold-standard measure, a reference sample — and quantify how tightly they track.
Check discrimination, not just correlation. Confirm the proxy responds to the target and not to something incidental it happens to correlate with.
Keep the calibration and reliability record. Log drift, error bands, and inter-source consistency over time, so detachment is caught as it happens rather than after a decision rests on it.

Tuning parameters¶

Criterion strength — how gold-standard the reference the proxy is checked against. A strong criterion gives a trustworthy verdict but is expensive or slow to obtain.
Audit cadence — one-off validation versus continuous monitoring. Continuous catches drift early but costs standing effort; one-off risks stale trust.
Validity threshold — how tight the proxy-target agreement must be to keep using the proxy. Strict protects decisions; lenient tolerates a noisier stand-in.
Scope of reliability — across time, across observers, across instruments. Widening the scope catches more failure modes but multiplies the checks.
Decommission trigger — how much decay retires the proxy versus merely flags it. Eager retirement protects truth; trigger-happy retirement churns the metric set.

When it helps, and when it misleads¶

Its strength is that it catches the most seductive failure in the whole archetype: a proxy that looks fine — stable, precise, cheap — while it has come loose from the target. This is the mechanism that operationalises the warning behind Goodhart's law, that a measure adopted as a target tends to stop being a good measure.^[1] Its reliability record also converts vague trust ("the sensor seems fine") into a dated, quantified statement.

Its own failure mode is being only as good as the criterion it audits against — if the "more direct" reference is itself a weak proxy, the audit launders one bad stand-in with another. It can also be run backwards, marshalled to defend a favoured metric by choosing a friendly criterion. And it validates the proxy's link, not the wisdom of measuring that target at all. The discipline is to anchor the audit to the strongest criterion affordable, to state the criterion's own limits, and to re-run it on a cadence matched to how fast the proxy can drift.

How it implements the components¶

instrument_or_proxy_translation_layer — takes the proxy-to-target mapping as its object and tests whether it still holds, rather than constructing it.
calibration_and_reliability_record — its standing output: the dated log of drift, error bands, and cross-source consistency that tracks the proxy's trustworthiness over time.

It does not build the readout-to-value rule in the first place (Instrument Readout Mapping), check agreement between independent observers (Intersubjective Replication Check), or assign the claim a semantic status (Operational Definition Table).

Instantiates: Sense-Experience Reduction Protocol — guards the archetype against proxies that have detached from the properties they were meant to make observable.
Consumes: Instrument Readout Mapping — supplies the proxy-to-value mapping this audit scrutinises.
Sibling mechanisms: Instrument Readout Mapping · Intersubjective Replication Check · Observation Protocol Design · Operational Definition Table · Disconfirming Condition Probe · Experience Prediction Matrix · Counterfactual Conditional Translation · Sensory Walkthrough · Phenomenological Interview · Access Boundary Annotation · Ethical Non-Production Rule

References¶

[1] "When a measure becomes a target, it ceases to be a good measure" (Marilyn Strathern's compact rendering of Goodhart's law). The audit is the standing check against exactly this drift: a proxy optimised for tends to detach from the property it once tracked. ↩