Versatile AI Risk Assessment

AI threat catalogue Model and Training Data Manipulation Development

Training Data Poisoning

Attackers inject manipulated or false data into the data an AI model learns from. The model picks up distorted patterns, becomes unreliable, or acquires hidden misbehavior that is very hard to spot afterwards.

As of: October 2026 · Catalogue version 2026.10.24.2 · 4 mitigations · 16 verified sources

Description

Many AI models learn from huge volumes of data collected automatically from public sources such as the internet, often without deep integrity checks. This is exactly where data poisoning strikes: attackers alter content in those sources, plant prepared examples, or take over expired internet domains that well-known training datasets still point to. Systems that keep learning from user input can be poisoned the same way while in live operation. The manipulation can target the initial training as well as the later fine-tuning of the model. Besides external attackers, insiders and already contaminated data deliveries from third parties are possible sources.

Possible impact

A poisoned model delivers degraded or deliberately skewed results that business teams rely on every day. This leads to bad decisions, quality defects, and reputational damage, for instance when the system produces discriminatory or false outputs. For high-risk AI, the EU AI Act explicitly names data poisoning as an AI-specific attack the system must be resilient against, so a lack of precautions also becomes a compliance risk.

Example

A company regularly retrains its spam filter on emails reported by users. Over several weeks, attackers report large numbers of prepared messages and gradually shift what the filter learns. Afterwards, the filter classifies the attackers' own phishing emails as harmless.

Recommended mitigations (4)

Compare effects and implementation levels. Expand each entry for the additional classification and rationale.

  • Data validation and sanitization

    Technical
    Effect
    Preventive
    Implementation level
    Data, Model & training
    Classification and rationale

    Primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.

  • Anomaly detection in training data

    Technical
    Effect
    Preventive, Detective
    Implementation level
    Data, Model & training
    Classification and rationale

    Primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.

  • Differential privacy (primarily privacy protection, limited effect against poisoning)

    Technical
    Effect
    Preventive
    Implementation level
    Data, Model & training
    Classification and rationale

    Primarily technical: Minimization, randomization, uniform processing, or query limits reduce information gained from repeated observations.

  • Robust training techniques

    Technical
    Effect
    Preventive
    Implementation level
    Data, Model & training
    Classification and rationale

    Primarily technical: A model, training, or data-processing method directly changes system behavior or robustness.

Framework mappings

Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.

OWASP LLM Top 10 LLM05:2026 · LLM06:2026NIST AI RMF Section 2.9 · MEASURE 2.5 · NISTAML.012 · NISTAML.013MITRE ATLAS AML.T0020EU AI Act Article 53(1)(d) · Article 55(1)(a) · Article 9(1), 9(2)(a), 9(2)(d)BSI R17 · R20 · R21BIML BIML-LLM LLMtop10:6 · BIML78 data:1 · BIML78 raw:2

Verified references (16)

Every reference states the framework, the exact location and the publishing organisation.

EU AI Act obligations related to this threat

The obligations from the obligations compass whose articles the catalogue maps to this threat. Each leads to its entry with core, action step and evidence; whether it applies to you depends on role and risk class.

  • Establish and operate a risk management system Art. 9 · applies from Annex III: 2 December 2027 · Annex I: 2 August 2028

    Statutory roles: Provider, Product manufacturer. Classes: High-risk under Annex III, High-risk as a product or safety component.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Draw up the technical documentation of the model Art. 53 Abs. 1 Buchst. a, Anhang XI · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: General-purpose AI model, Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Provide information to downstream providers Art. 53 Abs. 1 Buchst. b, Anhang XII · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: General-purpose AI model, Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Respect copyright and summarise the training content Art. 53 Abs. 1 Buchst. c und d · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: General-purpose AI model, Model with systemic risk.

    For models released under a free and open-source licence with publicly available parameters, the obligations under points (a) and (b) do not apply, but the copyright policy and the summary do. Models with systemic risk are excluded from this relief.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Evaluate and mitigate systemic risks Art. 55 · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

All 62 obligations by role, risk class and deadline in the obligations compass

Terms on this page

Glossary terms that occur in this entry. Every link leads to the full explanation.

  • Data Poisoning Manipulated training or reference data steers a model wrong on purpose.
  • Fine-tuning An existing model is trained further on your own data.

More entries from the topic group Model and Training Data Manipulation.

Targeted Poisoning / Label Poisoning Attackers tamper with a small, targeted portion of the training data, for example through wrong labels. Backdoor ML Model The model contains hidden behavior, a backdoor. It works correctly on normal inputs; only a secret trigger pattern in the input flips the … Sleepy Agent (Time/Event-Triggered Hidden Instructions) Malicious logic lies dormant inside the model and only activates later: on a set date, at a specific event, or in a particular environment.

Matching use cases

This list shows where to review this threat, rather than how high the risk is.

Particularly important for:

  • Recommender systemA seller creates artificial interest through purchased ratings and automated clicks. The model learns from this and prominently recommends the products to many people.
  • Dynamic pricingA competitor temporarily sets unusual prices or creates artificial demand. The model learns from this and adjusts its own prices in a way that harms the deployer.
  • Personalised online advertising (behavioural targeting)Automated click fraud creates apparent interest in particular advertisements. The model learns false relationships, and advertisers pay for nonexistent reach.
  • Algorithmic managementOperational measurements guide future allocations. Falsifying location data or deliberately rejecting jobs changes the values the system uses to assign work to everyone else.
  • Risk-based review selection for applications and benefitsThe cases reviewed and used for training are mainly those the model selected itself. This confirms earlier assumptions and increasingly targets groups already selected more often.
  • Predictive maintenanceTraining measurements pass through plant controllers and networks. Undetected manipulation can teach the model that a damaged condition is normal, preventing later warnings.
  • AI-based cyberattack detectionAn attacker who gains an early foothold gradually expands their activity until the system learns it as normal. Later data exfiltration then triggers no alert.

Typical for:

For 30 further use cases, relevance depends on the system design, for example whether it uses operational feedback or is trained on your own data. Show these use cases

Frequently asked questions

What is Training Data Poisoning?
Attackers inject manipulated or false data into the data an AI model learns from. The model picks up distorted patterns, becomes unreliable, or acquires hidden misbehavior that is very hard to spot afterwards.
Which mitigations help against Training Data Poisoning?
The catalogue lists 4 mitigations, each with control type, effect and implementation level. First among them are Data validation and sanitization, Anomaly detection in training data and Differential privacy (primarily privacy protection, limited effect against poisoning).
Which EU AI Act obligations does Training Data Poisoning touch?
The catalogue maps this threat to Art. 9, Art. 53 Abs. 1 Buchst. a, Art. 53 Abs. 1 Buchst. b, Art. 53 Abs. 1 Buchst. c und d and Art. 55 of the EU AI Act. In the obligations compass this leads to: Establish and operate a risk management system, Draw up the technical documentation of the model, Provide information to downstream providers, Respect copyright and summarise the training content and Evaluate and mitigate systemic risks. Whether an obligation applies depends on role, risk class and deadline.

Assess this threat in your own system

The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.

Start the live demo See the free modules

Cite this entry

For reports, policies or internal documents; the link leads directly to this entry.

“Training Data Poisoning”. Versatile AI Risk Assessment, AI threat catalogue, as of October 2026.
https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/training-data-poisoning/

← Back to the full catalogue