Skip to content

Ranked Leaderboard With Audit

Ranking-and-audit monitor — instantiates Bounded Rivalry Governance

Ranks entrants on a shared, published metric and audits the top of the board, so standing reflects real performance rather than whoever gamed the score best.

A Ranked Leaderboard With Audit makes performance continuously observable: it publishes a single comparable metric and a live ranking so every entrant sees where they stand and how far they trail. That visibility is the whole draw — but a naked leaderboard optimizes exactly what it measures, so it is paired with an audit that re-verifies entries and voids gamed ones. The defining idea is the coupling: maximal disclosure of the score, plus an integrity layer aimed precisely at the gaming that disclosure invites. Without the audit it amplifies cheating; without the disclosure it is just a private scoresheet.

Example

A public forecasting benchmark ranks models on accuracy against a shared dataset; anyone can submit and watch their rank move in real time. To stop teams from overfitting to the visible answers, the organizers keep a hidden test split and audit the top submissions — re-running the leaders' code on held-out data and disqualifying any entry that merely memorized the public set. The board stays motivating and comparable, but the top slots come to reflect genuine generalization rather than leaderboard-hacking. The moment the audit lapses, the summit fills with overfit entries.

How it works

  • Define one comparable metric and freeze how it is computed.
  • Publish standings so all entrants see rank and gap.
  • Audit the leaders — re-verify, spot-check, hold out data, or require reproducible submissions before crowning.
  • Void or penalize gamed and fabricated entries, then correct the board.

The distinguishing move is coupling maximal disclosure (a live public score) with an integrity check targeted at the gaming that public scores provoke.

Tuning parameters

  • Metric choice / composite — a single sharp metric is legible but gameable; a composite resists gaming but blurs the signal.
  • Public vs. hidden split — how much of the evaluation is visible; a held-out test set deters overfitting to the score.
  • Audit depth and frequency — a spot-check versus full reproduction of the top N; deeper audits cost more but catch more.
  • Update cadence — real-time versus periodic; faster feedback drives effort but amplifies gaming and churn.
  • Penalty for gaming — from a quiet correction to public disqualification.

When it helps, and when it misleads

Its strength is that a shared, visible metric aligns effort and lets everyone see the frontier, while the audit is what stops the board from simply rewarding whoever games hardest — it keeps the measure honest against Goodhart's law.[1]

Its failure mode follows from that law: any single metric is a proxy, and a public one becomes a target that pulls behavior toward the visible number and away from the real goal. Audits always lag the gaming they chase, and a live board can drive wasteful over-investment in rank for its own sake. Run backwards, the metric is chosen because a favored entrant already tops it. The discipline is to hold out part of the evaluation, audit the leaders before they are crowned, and revise the metric once it starts being gamed rather than defending a number that has stopped meaning anything.

How it implements the components

Ranked Leaderboard With Audit realizes the measurement-and-observability side of the archetype — how performance is scored and made visible:

  • performance_metric_and_scoring_basis — it defines and freezes the single comparable metric the whole field is ranked on, and audits it to keep it trustworthy.
  • information_disclosure_and_observability_rule — publishing live standings is its core act, making relative performance observable to every entrant.

It scores and exposes performance but does not set the penalties for sabotage or fouls (that's Sabotage or Foul Penalty Schedule) or cap what entrants may spend to climb (that's Spending Cap or Resource Cap).

Notes

A public running score is itself an incentive surface. Where the risk of gaming is high, a private/hidden split — entrants optimize against data they cannot see — keeps effort aimed at the true metric rather than the visible one, at the cost of the immediate feedback a fully public board provides.

References

[1] Goodhart's law — when a measure becomes a target, it ceases to be a good measure. A public leaderboard is the sharpest possible target, which is why the audit and a hidden test set are not optional extras but the very thing that keeps the score meaningful. ↩