AI threat catalogueModel and Training Data ManipulationDevelopment
Training Data Poisoning
Attackers inject manipulated or false data into the data an AI model learns from. The model picks up distorted patterns, becomes unreliable, or acquires hidden misbehavior that is very hard to spot afterwards.
Description
Many AI models learn from huge volumes of data collected automatically from public sources such as the internet, often without deep integrity checks. This is exactly where data poisoning strikes: attackers alter content in those sources, plant prepared examples, or take over expired internet domains that well-known training datasets still point to. Systems that keep learning from user input can be poisoned the same way while in live operation. The manipulation can target the initial training as well as the later fine-tuning of the model. Besides external attackers, insiders and already contaminated data deliveries from third parties are possible sources.
Possible impact
A poisoned model delivers degraded or deliberately skewed results that business teams rely on every day. This leads to bad decisions, quality defects, and reputational damage, for instance when the system produces discriminatory or false outputs. For high-risk AI, the EU AI Act explicitly names data poisoning as an AI-specific attack the system must be resilient against, so a lack of precautions also becomes a compliance risk.
Example
A company regularly retrains its spam filter on emails reported by users. Over several weeks, attackers report large numbers of prepared messages and gradually shift what the filter learns. Afterwards, the filter classifies the attackers' own phishing emails as harmless.
Recommended mitigations (4)
Every mitigation states its control type, effect, implementation level and the reason for the classification.
Data validation and sanitizationTechnical
- Effect
- Preventive
- Implementation level
- Data, Model & training
- Reason for the classification
- “Data validation and sanitization” is primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.
Anomaly detection in training dataTechnical
- Effect
- Preventive, Detective
- Implementation level
- Data, Model & training
- Reason for the classification
- “Anomaly detection in training data” is primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.
Differential privacyTechnical
- Effect
- Preventive
- Implementation level
- Data, Model & training
- Reason for the classification
- “Differential privacy” is primarily technical: Minimization, randomization, uniform processing, or query limits reduce information gained from repeated observations.
Robust training techniquesTechnical
- Effect
- Preventive
- Implementation level
- Data, Model & training
- Reason for the classification
- “Robust training techniques” is primarily technical: A model, training, or data-processing method directly changes system behavior or robustness.
Framework mappings
Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.
Verified references (15)
Every reference states the framework, the exact location and the publishing organisation.
- OWASP LLM Top 10 LLM04:2025 Data and Model PoisoningLLM04:2025 Data and Model Poisoning, official category page OWASP FoundationOriginal
- NIST AI RMF Section 2.9 Information SecuritySection 2.9, pp. 10–11 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF MEASURE 2.5 MEASURE 2.5MEASURE 2.5, p. 29 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF NISTAML.012 Clean-label PoisoningTaxonomy Index, p. x; Section 2.3.1, p. 20; related clean-label targeted attacks in Section 2.3.2, p. 21 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF NISTAML.013 Data PoisoningTaxonomy Index, pp. x–xi; Section 2.3.1, p. 19; Section 3.2.1, p. 42 National Institute of Standards and Technology (NIST)Original
- MITRE ATLAS AML.T0020 Poison Training DataATLAS.yaml technique object with id AML.T0020 (pinned release v5.6.0) MITREOriginal
- EU AI Act Article 53(1)(d) Obligations for providers of general-purpose AI modelsArticle 53(1)(d) European Union (EUR-Lex)Original
- EU AI Act Article 55(1)(a) Obligations of providers of general-purpose AI models with systemic riskArticle 55(1)(a) European Union (EUR-Lex)Original
- EU AI Act Article 9(1), 9(2)(a), 9(2)(d) Risk management systemArticle 9(1), 9(2)(a), 9(2)(d), read with Article 9(3) European Union (EUR-Lex)Original
- BSI R17 Vergiftung der Trainingsdaten (Data Poisoning) (Text, Bild, Video)Kap. 4, R17, p. 25 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BSI R20 Vergiftung über das Bewertungsmodell (Text, Bild, Video)Kap. 4, R20, p. 26 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BSI R21 Vergiftung über vorverarbeitende Komponenten (Text, Bild, Video)Kap. 4, R21, p. 27 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BIML BIML-LLM LLMtop10:6 Poison in the DataPDF p. 13, [LLMtop10:6:poison in the data] Berryville Institute of Machine Learning (BIML)Original
- BIML BIML78 data:1 PoisoningPDF p. 14, [data:1:poisoning] Berryville Institute of Machine Learning (BIML)Original
- BIML BIML78 raw:2 TrustworthinessPDF p. 10, [raw:2:trustworthiness] Berryville Institute of Machine Learning (BIML)Original
Terms on this page
Glossary terms that occur in this entry. Every link leads to the full explanation.
- Data Poisoning Manipulated training or reference data steers a model wrong on purpose.
- Fine-tuning An existing model is trained further on your own data.
Related threats
More entries from the topic group Model and Training Data Manipulation.
Assess this threat in your own system
The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.
Cite this entry
For reports, policies or internal documents; the link leads directly to this entry.
“Training Data Poisoning”. Versatile AI Risk Assessment, AI threat catalogue, as of July 2026. https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/training-data-poisoning/