Data Poisoning Attack

The adversarial-ML failure mode where an attacker corrupts training inputs so the model learns an attacker-chosen behaviour — an upstream contamination that runtime defenses are structurally powerless against and that clean benchmark accuracy cannot detect.

Version

v3 · 2026-09-28 · History · 5 corrections

Domain-specific #

244

Domain group

Applied Sciences & Engineering

Origin domain

Computer Science & Software Engineering

Subdomain

AI Security and Adversarial ML → Computer Science & Software Engineering

Core Idea

A data poisoning attack is the adversarial-ML failure mode in which an attacker corrupts the inputs to a learning process so the trained model's behaviour is shifted in an attacker-chosen direction at inference — often undetected at training time, since poisoned samples may be unremarkable and clean-benchmark accuracy stays high. It is upstream of inference, targeting the training pipeline itself. Variants include label-flipping, clean-label backdoors with imperceptible triggers, availability attacks, and federated-learning gradient poisoning.

How would you explain it like I'm…

The Sneaky Picture Book

Imagine a robot learning animals from a picture book. A sneaky person slips in a few pages where cats are labeled "dog," or where anything with a tiny sticker is labeled "dog." The robot seems to learn fine, but later it makes the mistakes the sneaky person wanted.

Poisoning the Learning Examples

Computers that learn, called machine learning models, study lots of examples to learn how to do a job. A data poisoning attack is when someone secretly mixes bad examples into what the computer studies. The bad examples might look normal one by one, and the computer might still score well on its usual tests. But afterward it behaves the way the attacker wanted: getting certain things wrong, getting worse at everything, or reacting to a secret hidden signal. This is different from tricking a computer that has already finished learning, because here the trick happens while it is still learning.

Training-Data Corruption Attack

A data poisoning attack is an attack on machine learning where someone corrupts the training data so the finished model behaves the way the attacker wants when it's used. It happens before the model is deployed, which makes it different from adversarial examples (tricky inputs made at run time against a finished model) and from data extraction (questioning a model to pull out private information). It can go unnoticed because each poisoned example may look ordinary and the model may still score well on clean tests. Types include label flipping (changing some correct labels to wrong ones), backdoor attacks (hiding a trigger pattern that causes a chosen mistake while the model otherwise acts normally), availability attacks (making the model worse for everyone), and poisoning in federated learning, where one participant sends bad updates. Because huge models learn from internet-scale data that no one can fully check, they are especially exposed.

 

A data poisoning attack is an adversarial machine-learning failure mode in which an attacker corrupts the inputs to training so that the resulting model's inference-time behavior shifts in an attacker-chosen direction. It may evade training-time detection because poisoned samples can be individually unremarkable and clean-benchmark accuracy can stay high. It is upstream of inference: unlike adversarial examples, which are crafted at runtime against a fixed model, or data extraction, which queries a deployed model, poisoning targets the training pipeline itself, exploiting the pipeline's trust in its data sources and the inheritance of training-time properties by every downstream deployment. Label-flipping relabels a fraction of correct samples to wrong classes, shifting decision boundaries; it is relatively detectable by outlier inspection but effective in aggregate. Clean-label backdoor attacks, building on triggerless clean-label poisoning by Shafahi and colleagues and developed in the label-consistent attacks of Turner, Tsipras, and Madry and Saha and colleagues' Hidden Trigger Backdoor work, plant correctly labeled, benign-looking samples carrying an imperceptible spatial, spectral, or stylistic trigger, so triggered inputs are misclassified to a target while other inputs behave normally. Availability attacks aim at general performance degradation for all users. In federated learning, a participating client submits malicious gradient updates that corrupt the jointly trained model without exposing the poisoning data to a central auditor. For large language models, poisoned web pages in pretraining corpora, planted falsehoods, and jailbreak-enabling samples in RLHF preference data instantiate the same structure at a scale where comprehensive auditing is not tractable.

Scope of Application

Data poisoning lives across the subfields of adversarial machine learning, within training-time integrity.

• Supervised-learning red-teaming — poisoning-fraction threat models and benchmark attacks (BadNets, Hidden Trigger).
• LLM training-set contamination — poisoned web pages, false facts, and jailbreak samples in RLHF data.
• Federated learning — a client submitting malicious gradient updates, met by Byzantine-robust aggregation.
• Recommendation and ranking systems — review fraud and shill attacks poisoning the learned signal.
• Spam and abuse classifiers — borderline content submitted to drift the decision boundary over time.

Clarity

Naming data poisoning fixes the pipeline stage where the attack lives, making the defensive landscape legible. A model can misbehave from a runtime adversarial example, from extraction, or because it learned the wrong function first — three failures that look alike but yield to different controls. Holding poisoning as the training-time case tells the defender the entire inference-time toolkit is structurally powerless against it. Its second contribution is breaking the assumption that benchmark performance certifies integrity: the failure is silent.

Manages Complexity

The named attacks differ on every visible axis, and a defender reasoning attack by attack faces a fast-multiplying catalogue. Naming the data poisoning attack collapses that zoo onto a single locus: all are contamination of the training substrate, upstream of inference. The defender tracks one stage and one trust relation, from which the impotence of runtime defenses, the location of effective upstream ones, and the unreliability of benchmark accuracy all follow without re-derivation per variant.

Abstract Reasoning

The concept licenses diagnostic reasoning (inferring a training-time origin from a misbehaviour runtime monitoring can't see, and reading a behavioural signature back to a variant), interventionist reasoning (moving defense upstream, predicting runtime controls are futile), boundary-drawing (separating poisoning from sibling attacks by stage, "behaves correctly on the eval distribution" from "contains no attacker-chosen behaviour"), and prediction (silent failure, small-fraction efficacy, and the intractability of auditing at scale).

Knowledge Transfer

Within adversarial ML the concept transfers as mechanism, intact — the training-time diagnostic, signature-to-variant attribution, upstream-defense prescription, and silent-failure predictions carry across supervised red-teaming, LLM contamination, federated learning, and recommender fraud. Beyond ML the pattern — contaminate the substrate a system learns from — recurs vividly (propaganda, fake reviews, Batesian mimicry, benchmark gaming), and precisely because none would be called "data poisoning," the AI name should not carry it. The travelling primes are Learning-Substrate Contamination (the declared parent), feedback_loop, Selective Propagation, and Trust.

Relationships to Other Abstractions

Current abstraction Data Poisoning Attack Domain-specific

Parents (1) — outgoing typed relations

• Data Poisoning Attack is a decomposition of Learning-Substrate Contamination Prime

  Data poisoning is learning-substrate contamination specialized to ML training data, labels, gradients, triggers, and training-pipeline defenses.

Children (1) — abstractions linked here by typed relations

• Model Skewing Domain-specific is a kind of Data Poisoning Attack

  Model skewing is data poisoning specialized to a clean-launched, continuously or periodically updating deployment whose production stream is biased gradually after launch.

Hierarchy paths (2) — routes to 2 parentless roots

• Data Poisoning Attack → is a decomposition of Learning-Substrate Contamination → presupposes Learning → is a kind of Adaptation

Show alternative path (1)

Neighborhood in Abstraction Space

Data Poisoning Attack sits in a sparse region of the domain-specific corpus (65^th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Adversarial AI & Security Attacks (14 abstractions)

Nearest neighbors

• Model Skewing — 0.86
• Data Extraction Through Prompting — 0.85
• AI Supply-Chain Attack — 0.85
• Prompt Injection — 0.84
• Model Inversion Attack — 0.84

Computed from structural-signature embeddings · 2026-10-25