Benchmark Deduplication Scan¶
Benchmark audit — instantiates Leakage-Resistant Validation Design
Searches the training and development corpus for copies or restatements of the evaluation benchmark, so a memorised answer can't masquerade as a solved problem.
When a model is scored on a public benchmark, the benchmark's own questions and answers may already sit inside the training corpus — scraped, mirrored, or discussed on the open web. The Benchmark Deduplication Scan searches the training and development material for those items and records which benchmark cases the model has effectively already seen. Its distinguishing feature is direction and asymmetry: unlike a within-dataset duplicate scan, the reference is a specific, named external benchmark, and the haystack is a vast, mostly-unlabelled training corpus. It answers a question no accuracy figure can — "is this score capability, or recall?" — and produces a register of exposure that lets the number be reported honestly.
Example¶
A team evaluates a language model on a public multiple-choice benchmark and posts an impressive score. Before trusting it, they run a benchmark deduplication scan: they take each benchmark item and search the pretraining corpus for verbatim and near-verbatim occurrences — the question stem, the answer options, characteristic phrasings — using long n-gram overlap so incidental word matches don't fire. A meaningful slice of items turn up in scraped forum threads and study-guide pages, some with the correct answer stated outright. Those items are logged to an exposure register and excluded from the reported score, which is republished as a contamination-adjusted figure. The headline number falls, but it now measures reasoning on unseen items rather than recall of memorised ones.
How it works¶
Its distinctive machinery is one-directional search from a known benchmark into an uncontrolled corpus:
- Take the benchmark items as the query set and search the training (and any retrieval or tool-output) corpus for matches.
- Use overlap measures robust to a huge corpus — long n-gram overlap or substring hashing — so common phrasing doesn't trigger false hits, and optionally planted canary strings to detect ingestion of pre-marked sets.
- Score each benchmark item's exposure — none, partial (the question but not the answer), or full — and record it.
- Remediate by either dropping exposed items from the reported score or purging the matches from training and re-evaluating (decontamination).
Tuning parameters¶
- Match granularity — exact string vs. n-gram overlap fraction vs. embedding similarity. Longer n-grams cut false positives but miss paraphrase and translation of the same item.
- Corpus coverage — training data only, or training plus the retrieval index and tool outputs. A benchmark can leak at inference through retrieval even if pretraining was clean.
- Canary vs. blind discovery — rely on pre-planted canary strings, or search blind. Canaries only catch benchmarks you thought to mark in advance.
- Exposure threshold — how much overlap marks an item "contaminated" and pulls it from the reported score.
- Remediation mode — exclude exposed items from the metric, or purge them from training and retrain — cheaper reporting vs. a genuinely uncontaminated model.
When it helps, and when it misleads¶
Its strength is that it separates memorisation from skill on exactly the public benchmarks whose scores travel furthest, and it lets a team publish a contamination-adjusted number instead of an inflated headline.
It misleads through what string search cannot see. Paraphrase, translation, reformatting, and solution discussion evade verbatim matching, so a clean scan is weak evidence of a clean corpus — absence of a match is not absence of exposure. It is also powerless over corpora you don't control, which is most closed pretraining data. The classic misuse is to scan only for exact strings, declare the benchmark clean, and publish. The discipline is to treat the scan as a lower bound on contamination[n1], report the method and its granularity alongside the score, and prefer freshly built or held-out benchmarks when the corpus is unknowable.
How it implements the components¶
benchmark_exposure_register— its primary output: a ledger of which benchmark items were found in the training or retrieval corpus and at what exposure level.contamination_probe_suite— the corpus search is the suite's benchmark-overlap probe, aimed specifically at leakage of a named external evaluation set.
It builds the exposure ledger but does not itself score how much the exposure distorts capability — that impact rating is the Leakage Ablation Test — and it targets external-benchmark overlap, not within-dataset near-duplicates, which is Duplicate and Near-Duplicate Scan.
Related¶
- Instantiates: Leakage-Resistant Validation Design — the scan checks the boundary between training corpus and evaluation benchmark.
- Sibling mechanisms: Duplicate and Near-Duplicate Scan · Leakage Ablation Test · Holdout Access Log · Fresh Holdout Retest · Feature Availability Audit · As-Of Join Rule · Entity-Grouped Split · Time-Based Holdout · Nested Cross-Validation · Preprocessing Fit-on-Training-Only · Label Proxy Screen
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Searches the training and development corpus for copies or restatements of the evaluation benchmark, so a memorised answer can't masquerade as a solved problem, making its operative form a bounded evaluation of existing evidence or work that produces a finding or disposition.
Independent corroboration: The frozen evidence defines Benchmark Deduplication Scan as 'Searches the training and development corpus for copies or restatements of the evaluation benchmark, so a memorised answer can't masquerade as a solved problem', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Single lineage
Present-day reach: Specialized
Rationale: Machine-learning evaluation developed benchmark decontamination scans that search training corpora for evaluation-item overlap so memorization is not reported as general capability.
Related originating lineages:
- Data Science & Analytics — Dataset governance operationalizes corpus inspection, exclusion registers, and contamination-adjusted reporting.
Review resolution: The GPT-3 paper describes producing clean benchmark versions by removing examples with 13-gram overlap against pretraining data, while later primary research shows why paraphrase and translation evade simple string matching. That is precisely this one-directional scan and its stated limitation, placing the mechanism in machine-learning computer science with data governance as an alternate.
Attribution caveat: Computer science and data science overlap strongly here; the mechanism's n-gram, hashing, and approximate-retrieval techniques make computer science the narrower origin label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
- Language Models are Few-Shot Learners — Benchmark Contamination Analysis
- Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
Notes¶
The scan reports a floor, not a verdict. Because paraphrase and unseen mirrors escape matching, a benchmark that scans clean can still be partly memorised — so the exposure register is best read as "at least this much contamination," and a suspiciously high score on a scanned-clean public benchmark still warrants a fresh, never-published test set rather than relief.
[n1] Benchmark (train-test) contamination — the leakage of evaluation items into training data, addressed in practice by n-gram-overlap decontamination and canary strings. String-based scanning detects verbatim and near-verbatim exposure and therefore establishes a lower bound on the true contamination. ↩