Skip to content

Contest or Autograder Harness

Evaluation harness — instantiates Generate-and-Verify Separation

Lets many untrusted entrants submit candidates through a fixed contract and scores each automatically against a held-out test suite, so anyone can compete without being trusted.

Contest or Autograder Harness is automated judging machinery built for scale and openness: it opens submission to a crowd of untrusted generators — contestants, students, competing vendors — and mechanically scores each submission against a battery of held-out tests, with no human reviewer in the accept loop. The distinguishing idea is that its whole reason to exist is to let strangers participate safely: because a fixed harness, not a trusted person, renders the verdict, the number of generators can grow without the trust burden growing with it. Its authority, and its limits, come from one thing — the test suite. A submission is "accepted" exactly insofar as it passes the tests the harness happens to hold.

Example

An online programming-contest judge hosts a problem that a few thousand entrants attempt. Each submits source code through a strict contract — a named entry point, fixed input/output format on stdin/stdout, a time limit of a few seconds and a memory cap. The harness compiles each submission, runs it in isolation against a battery of hidden test cases (tiny edge cases, maximum-size stress inputs, adversarial ones), and accepts only submissions that produce the exact expected output on every test within the limits — then ranks the accepted ones by runtime. Nobody vetted the entrants; a submission from an unknown account is treated identically to any other, because the judge, not a reviewer, decides. The result set is trustworthy to the exact degree the hidden tests are, and no further.

How it works

  • Open submission, one judge. Any number of untrusted entrants submit; a single automated harness evaluates all of them by the same rule, which is what makes open participation safe rather than reckless.
  • A contract at the door. Every submission must conform to a fixed interface — entry point, I/O format, resource limits — so malformed candidates are rejected mechanically before scoring.
  • Score against a held-out proxy. Acceptance is decided by a battery of tests the entrant does not fully see; the verdict is a sample of correctness, and ranking (speed, score, partial credit) layers on top.

Tuning parameters

  • Test coverage & secrecy — how much of the suite is hidden and how thoroughly it probes edge and stress cases. More hidden coverage makes overfitting harder but starves entrants of feedback.
  • Scoring function — binary all-or-nothing acceptance, partial credit per test, or a ranked score. This sets what entrants optimize toward.
  • Submission-contract strictness — how rigid the format and limits are; strict contracts reject malformed candidates cheaply but raise the barrier to entry.
  • Resource limits — time and memory caps, which turn efficiency into part of the acceptance criterion rather than just correctness.
  • Anti-gaming limits — submission-rate throttling and result opacity, to deter probing the hidden tests by brute force.

When it helps, and when it misleads

Its strength is throughput and openness: it judges a crowd cheaply, repeatably, and identically, letting talent or solutions come from anyone without a trust relationship first. That is the asymmetry paying off — many hard-to-produce candidates, one cheap-to-run judge.

Its failure mode is that passing the tests is not being correct. A wrong solution that happens to satisfy every held-out case is accepted, and entrants who can see or infer the tests will tune to them — Goodhart's law in action, the measure degrading the moment it becomes the target.[1] The classic misuse is teaching-to-the-test: optimizing submissions (or a model) to the leaderboard rather than the real objective the tests were meant to stand for, sometimes abetted by a hidden-test leak. The discipline is to hold tests out, rotate and expand them, and treat the score as a coverage-bounded estimate of quality, never a proof of it.

How it implements the components

The harness fills the openness-and-scoring components — the parts a public judge owns:

  • generator_verifier_role_separation — it cleanly separates the many untrusted entrants (generators) from the single automated judge (verifier), the split that lets strangers compete without being trusted.
  • candidate_claim_contract — the submission contract fixes the format, entry point, and resource limits every candidate must meet even to be scored.
  • coverage_and_completeness_boundary — the held-out test suite is an incomplete proxy for correctness, and the harness's guarantee is explicitly bounded by what those tests cover.

It does not itself isolate untrusted code from the host — that is Sandboxed Candidate Execution, which it relies on — and it does not set stakes or penalties on entrants, which is the Fraud-Proof Challenge Protocol.

Notes

The harness certifies "passed our tests," never "solved the problem." Strengthening it means improving coverage — which is why the coverage boundary is its own component and not a footnote — and why two harnesses on the same problem can disagree simply because their test suites do.

References

[1] Goodhart's law — "when a measure becomes a target, it ceases to be a good measure." A held-out test suite is a proxy for correctness; the moment entrants optimize for the proxy rather than the goal, the proxy's value as evidence erodes, which is why hidden and rotating tests are standard practice. ↩