Investigating Data Contamination in Modern Benchmarks for Large Language Models.¶
Deng, C., Zhao, Y., Tang, X., Gerstein, M., & Cohan, A. (2024). Investigating Data Contamination in Modern Benchmarks for Large Language Models. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 8706-8719.
Cited by¶
1 citation across 1 artifact.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Data Leakage
- And in adversarial settings, published test benchmarks eventually leak into the training data of the systems they were meant to evaluate.
This sourceDocuments that published evaluation benchmarks leak into the pretraining corpora of the LLMs they were meant to evaluate, inflating reported scores — the adversarial benchmark-contamination case of data leakage.
Supported in partVerified against the publisher's abstract
Documents benchmark contamination inflating reported LLM scores — backing the adversarial-benchmark clause only; the sentence's finance look-ahead-bias clause is outside this paper's scope.
“Recent observations have underscored a disparity between the inflated benchmark scores and the actual performance of LLMs, raising concerns about potential contamination of evaluation benchmarks.”
- And in adversarial settings, published test benchmarks eventually leak into the training data of the systems they were meant to evaluate.
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 1 of 1 citation: 1 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:76fbe383f4fb · see in the full table