Poisoning Web-Scale Training Datasets is Practical.¶
Carlini, N. (2024). Poisoning Web-Scale Training Datasets is Practical. 2024 IEEE Symposium on Security and Privacy (S&P), 407-425.
Cited by¶
2 citations across 2 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Inherited-Substrate Risk
- AI transfer learning: a fine-tuned model inherits the foundation model's training-data biases, prompt injections, and implanted triggers; the downstream team audited their fine-tuning data, not the foundation, so the risk lives across the boundary.
This sourceDemonstrates that foundation-model training corpora can be poisoned upstream, so fine-tuned models inherit implanted vulnerabilities.
- AI transfer learning: a fine-tuned model inherits the foundation model's training-data biases, prompt injections, and implanted triggers; the downstream team audited their fine-tuning data, not the foundation, so the risk lives across the boundary.
Domain-specific¶
- Data Poisoning Attack
- In the large-language-model setting, poisoned web pages seeded into pretraining corpora, false information planted to be learned as fact, and jailbreak-enabling samples embedded in RLHF preference data are all instantiations of the same upstream-contamination structure applied to internet-scale data pipelines where comprehensive auditing is not tractable
This sourceDemonstrations that web-scale training corpora, including the Wikipedia snapshots used for language-model pretraining, can be poisoned practically because datasets of that size cannot be manually curated; RLHF preference-data poisoning is not covered. The demonstration that real web-scale training datasets such as LAION-400M could be poisoned in practice for as little as $60. Frontrunning poisoning, in which malicious Wikipedia edits are timed to land just before a predictable dataset snapshot is taken.
Supported in partVerified against the source
- In the large-language-model setting, poisoned web pages seeded into pretraining corpora, false information planted to be learned as fact, and jailbreak-enabling samples embedded in RLHF preference data are all instantiations of the same upstream-contamination structure applied to internet-scale data pipelines where comprehensive auditing is not tractable
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 1 of 2 citations: 1 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:6821f9c584ec · see in the full table