A machine learning model learns patterns from its training data. If an attacker can change that data, they can change what the model learns. The US National Institute of Standards and Technology (NIST) treats this as one of the core categories of attack on AI systems. In its taxonomy of adversarial machine learning report, NIST says poisoning attacks occur in the training phase, when corrupted data is introduced. That separates them from evasion attacks, which tamper with inputs after a system is deployed. NIST’s examples, as relayed by Data Center Knowledge, include slipping inappropriate language into conversation records so a chatbot treats it as normal.

NIST’s report, authored by Apostol Vassilev, Alina Oprea, and colleagues, covers both predictive AI (classifiers, recommenders) and generative AI (chatbots, image generators). It also separates data poisoning, which corrupts the dataset, from model poisoning, which tampers with the model itself.

Read also: What are data poisoning attacks?

 

Two outcomes

Poisoning generally aims at one of two outcomes.

Degradation attacks try to make a model worse overall. The attacker floods the training set with noise, mislabeled examples, or biased material so accuracy drops.

Backdoor attacks are more subtle. The model behaves normally almost all the time but does something specific when it sees a hidden trigger, such as a rare phrase, a pixel pattern, or a particular image. NIST’s report lists “backdoor attack” and “trojan attack” among its key terms. Backdoors are hard to catch because ordinary testing never fires the trigger.

NIST also distinguishes attackers by how much they know about the target. Nextgov’s coverage of the report describes white-box attackers, who have near-complete knowledge of the system, black-box attackers, who have little to none, and gray-box attackers in between.

 

Why it’s so easy to attempt

Modern AI models are trained on data scraped from the open internet. Nobody can manually review billions of web pages, so anyone who can publish content online has a possible path into a training set.

A team led by Nicholas Carlini, with researchers including Florian Tramèr and Hyrum Anderson, showed this is practical and not just theoretical. Their paper “Poisoning Web-Scale Training Datasets is Practical” describes attacks that the authors call “immediately practical and could, today, poison 10 popular datasets.” They demonstrated two techniques:

  • Split-view poisoning exploits the fact that internet content changes. A dataset is often just a list of URLs, so what an annotator saw when the list was built can differ from what a later user downloads.
  • Frontrunning poisoning targets datasets built from periodic snapshots of crowd-sourced sites like Wikipedia. An attacker only needs a time-limited window to inject bad content before the snapshot is taken.

The authors notified the maintainers of the affected datasets and recommended several low-overhead defenses.

Government agencies have taken up the same concern. In May 2025 the NSA’s Artificial Intelligence Security Center, CISA, the FBI, and agencies from Australia, New Zealand, and the UK issued joint guidance titled AI Data Security: Best Practices for Securing Data Used to Train & Operate AI Systems. SC Media’s summary says the guidance describes both split-view and frontrunning poisoning and notes that expired domains in AI datasets can be taken over by malicious actors. According to law firm Davis Wright Tremaine’s analysis, it treats web-scale datasets as especially exposed to poisoning, disinformation, and outdated content.

The guidance also separates poisoning from “data drift,” the gradual degradation that happens as real-world conditions change. SC Media reports that poisoning tends to occur more quickly and suddenly than drift, which unfolds slowly.

 

How little poison does it take?

A 2025 study by the UK AI Security Institute, the Alan Turing Institute, and Anthropic trained language models from 600 million to 13 billion parameters. Engadget reported that a small and fairly constant number of malicious documents could poison a model regardless of its size or training data, with 250 documents enough to implant a backdoor, “a much smaller number than expected.” Coverage of the study put those 250 documents at roughly 0.00016% of the largest model’s training data. In the experiments, 100 poisoned documents were not enough to reliably backdoor any model, while 250 or more worked across all sizes.

Furthermore, in healthcare, a study in Nature Medicine simulated a poisoning attack on The Pile, a popular dataset for LLM development. The researchers found that replacing “just 0.001% of training tokens with medical misinformation” produced models more likely to propagate medical errors. The corrupted models matched clean ones on open-source benchmarks routinely used to evaluate medical LLMs, so standard tests missed the damage. Senior author Eric Oermann, a neurosurgeon at NYU Langone, explained the root issue to the health system’s Physician Focus publication: “LLMs heavily rely on unverified training data.”

 

Poisoning as a tool for creators

Data poisoning isn’t only a weapon against AI developers. Some people use it defensively.

In 2023, researchers at the University of Chicago led by Professor Ben Zhao released Nightshade, a tool that lets artists make invisible changes to pixels before posting their work online. MIT Technology Review reported that if such images are scraped into a training set, they can cause the resulting model to break unpredictably, so that “dogs become cats, cars become cows.” The magazine also reported that the effect spreads across related concepts: a poisoned “dog” also affects “puppy,” “husky,” and “wolf.”

The team’s preprint states that “data poisoning attacks manipulate training data to introduce unexpected behaviors,” according to The Daily Beast’s coverage. Zhao framed Nightshade as a way to tip the power balance back toward artists, who say opt-out policies from AI companies aren’t enough.

 

Real-world stakes

Where could poisoning do the most harm?

  • Healthcare: corrupted medical models could spread dangerous misinformation while looking fine on benchmarks.
  • Cybersecurity: the NSA/CISA guidance, as summarized by Davis Wright Tremaine, notes that where AI is used to detect and stop cyber threats, poisoning can make attacks harder to detect.
  • Autonomous systems and finance: NIST-related analysis points to sectors like driving and finance where adversarial attacks can cause real disruption.
  • Public information: chatbots that absorb planted falsehoods can repeat them at scale.

 

Defenses

NIST acknowledges there is “no silver bullet,” and Vassilev noted that the defenses devised so far are incomplete. Commonly recommended measures include:

  1. Data sanitization and provenance tracking. ExecutiveGov’s coverage of NIST’s report lists sanitizing data, modifying training algorithms, and robust training as poisoning mitigations. The NSA/CISA guidance notes tracking provenance and verifying integrity in storage and transit, with digital signatures as one tool.
  2. Dataset integrity checks. The Carlini team recommended low-overhead defenses such as cryptographic hashes to detect when content at a URL changes after curation.
  3. Monitoring and validation. Summary of NIST’s guidance includes watching performance metrics for large drops and augmenting training with correctly labeled adversarial examples.
  4. Output screening. The Nature Medicine study’s knowledge-graph screening shows how domain checks can catch harmful outputs even from a compromised model.
  5. Security hygiene across the AI lifecycle. The joint agency guidance recommends threat modeling at the planning stage, trusted infrastructure for training data, and updating incident response plans to cover AI-specific threats such as poisoning.

Read also: What is model poisoning?

 

FAQs

Can data poisoning affect any type of AI system?

Yes, it can target both predictive AI (like classifiers and recommenders) and generative AI (like chatbots and image generators).

 

Who carries out data poisoning attacks?

Attackers range from those with near-complete knowledge of a system to those with almost none, and some creators also use poisoning defensively to protect their work.

 

Why is data poisoning hard to detect?

Poisoned models can perform normally on standard tests, so the damage often stays hidden until a trigger or specific situation exposes it.