Skip to content

Fitness Proxy Audit

Audit — instantiates Adaptive Barrier-Circumvention Response

Audits what your barrier and its metrics actually reward for surviving — exposing proxies that let an escape variant look 'handled' precisely because it has become harder to see.

Every barrier and every metric quietly defines a fitness function: whatever survives, spreads, or scores well is what the system is really selecting for. Fitness Proxy Audit treats that function as the suspect. Rather than measuring the population (a monitor) or mapping which control covers which class (a matrix), it interrogates the proxies themselves — the outcome, payoff, survival, and detectability signals the barrier and its dashboards implicitly reward — and asks the one question the other mechanisms assume away: are we rewarding safety, or merely rewarding invisibility? Its defining move is hunting for escape classes that look successful only because observation is weak, so a metric can improve while the world it measures gets worse.

Example

A platform's trust-and-safety team audits its abuse controls. The headline numbers look excellent: reported-abuse removal is high and proactively detected prevalence is trending down. The audit inventories the implicit proxies behind those numbers — the system rewards content that is neither reported nor matched by the classifier — and then asks whether any escape class scores well purely by being unobservable. It finds a candidate: abuse drifting into formats the classifier parses poorly, and into private channels where reporting is rare, would register as prevalence down while actually reflecting detectability down.

That is the crux — the prevalence proxy silently conflates "less abuse" with "less-detectable abuse," a coverage decay dressed up as success. The pattern is textbook Goodhart's law: once the measure became the target, it stopped measuring the thing. The audit's output is corrective, not just diagnostic: add an observation channel that does not share the classifier's blind spot (sampled human review of low-signal formats), and split the metric so detectable and hard-to-detect classes are reported separately rather than averaged into a reassuring single line.

How it works

  • Inventory explicit and implicit proxies. List everything the barrier and monitoring actually reward for surviving — especially the detectability proxies nobody wrote down.
  • Compare proxies to the real objective. Locate where "scores well" diverges from "is genuinely acceptable."
  • Probe for harmful invisibility. Spend independent observation specifically to test whether a falling number means the class is gone or merely hidden.
  • Revise the target. Correct the metric, the monitoring channel, or the pressure allocation wherever a proxy is misaligned — the deliverable is a fixed measurement, not a memo.

Tuning parameters

  • Proxy-inventory depth — how hard you hunt for implicit proxies. Deeper hunting finds subtle invisibility but costs analyst time.
  • Objective specification — how precisely the "real" goal is stated. A vague objective makes misalignment unprovable and lets the audit be waved off.
  • Invisibility-probe budget — how much independent observation you spend to distinguish "down" from "hidden." This is the audit's load-bearing dial.
  • Reporting granularity — one aggregate score versus a breakdown by detectability class. Decomposition reveals decay but complicates the dashboard.
  • Cadence — one-time versus periodic. Proxies drift as the population adapts, so a single audit stales quickly.

When it helps, and when it misleads

Its strength is catching the specific, insidious failure the rest of the loop is blind to: suppression that looks like success because the survivors merely went quiet — the metric climbing while protection erodes. By correcting the target, it protects every downstream monitor and gate from optimizing a corrupted proxy.

Its failure mode is that the audit needs some independent observation to confirm invisibility; with none, it can only raise suspicion, not settle it. Its classic misuse is being run backwards — marshalled to defend a favored metric rather than to test it, declaring a proxy sound because no one went looking for counter-evidence.[1] The discipline is to treat "no signal" as possibly "no sensor," and to spend real observation against the invisibility hypothesis before concluding the barrier still works.

How it implements the components

  • selection_pressure_profile — it reconstructs what the barrier actually selects for — which survival, payoff, and detectability traits are rewarded — which is exactly this profile read through the lens of the system's proxies.
  • coverage_decay_indicator — it exposes population-weighted protection decay that a headline metric is hiding, flagging when "measure up" masks "coverage down."

It does not map per-control coverage (that's Barrier Coverage Matrix), track live composition (that's the Variant-Composition Surveillance Dashboard), or aggregate system-wide net harm (that's System-Wide Net-Risk Dashboard). This mechanism audits the measurement itself; siblings measure the world with it.

References

[1] Goodhart's law — "when a measure becomes a target, it ceases to be a good measure" — is the exact hazard here: an adaptive population optimizes against whatever proxy the barrier scores on, so the proxy that once tracked harm ends up tracking only detectable harm. ↩