Versatile AI Risk Assessment

AI threat catalogue Model and Training Data Manipulation Development

Backdoor ML Model

The model contains hidden behavior, a backdoor. It works correctly on normal inputs; only a secret trigger pattern in the input flips the output to whatever result the attacker has chosen.

As of: October 2026 · Catalogue version 2026.10.24.2 · 4 mitigations · 16 verified sources

Description

To plant a backdoor, attackers tie an inconspicuous trigger pattern, such as a specific image element or character sequence, to an output of their choosing. The pattern can be designed so that humans never notice it. The backdoor enters the model through poisoned training data, directly altered model weights, or compromised pre-trained models from public sources. Such backdoors can persist even when the organization later retrains the model or hardens it with additional safety training. Because the model behaves correctly on all normal inputs, standard testing rarely uncovers a backdoor.

Possible impact

The attacker can trigger the misbehavior at any time and thereby disable security and screening functions such as access controls or detection systems. From the moment of activation, the system's results and automated decisions can no longer be trusted. The organization faces security incidents, contract breaches, and, for high-risk AI, regulatory consequences because the robustness required there is missing.

Example

An office building controls entry with an AI camera meant to detect dangerous objects. A backdoor was planted in the purchased model: anyone wearing a garment with a specific print passes without an alarm, even while visibly carrying a weapon.

Recommended mitigations (4)

Compare effects and implementation levels. Expand each entry for the additional classification and rationale.

  • Model scanning for backdoors

    Technical
    Effect
    Detective
    Implementation level
    Model & training
    Classification and rationale

    Primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.

  • Neural cleanse techniques

    Technical
    Effect
    Detective
    Implementation level
    Model & training
    Classification and rationale

    Primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.

  • Activation clustering analysis

    Technical
    Effect
    Detective
    Implementation level
    Model & training
    Classification and rationale

    Primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.

  • Train from trusted base models only

    Organizational & process-based
    Effect
    Preventive
    Implementation level
    Model & training, Supply chain
    Classification and rationale

    Complementary control typeTechnical

    Primarily organizational and process-based: Defined selection, operating, or lifecycle procedures make the control binding and repeatable; complemented by technical implementation.

Framework mappings

Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.

OWASP LLM Top 10 LLM04:2026 · LLM05:2026NIST AI RMF Section 2.9 · MEASURE 2.7 · NISTAML.021 · NISTAML.023 · NISTAML.026 · NISTAML.051MITRE ATLAS AML.T0018EU AI Act Article 53(1)(a) · Article 55(1)(a) · Article 9(1), 9(2)(a), 9(2)(d)BSI R19 · R20BIML BIML-LLM model:4 · BIML78 model:2

Verified references (16)

Every reference states the framework, the exact location and the publishing organisation.

EU AI Act obligations related to this threat

The obligations from the obligations compass whose articles the catalogue maps to this threat. Each leads to its entry with core, action step and evidence; whether it applies to you depends on role and risk class.

  • Establish and operate a risk management system Art. 9 · applies from Annex III: 2 December 2027 · Annex I: 2 August 2028

    Statutory roles: Provider, Product manufacturer. Classes: High-risk under Annex III, High-risk as a product or safety component.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Draw up the technical documentation of the model Art. 53 Abs. 1 Buchst. a, Anhang XI · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: General-purpose AI model, Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Provide information to downstream providers Art. 53 Abs. 1 Buchst. b, Anhang XII · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: General-purpose AI model, Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Respect copyright and summarise the training content Art. 53 Abs. 1 Buchst. c und d · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: General-purpose AI model, Model with systemic risk.

    For models released under a free and open-source licence with publicly available parameters, the obligations under points (a) and (b) do not apply, but the copyright policy and the summary do. Models with systemic risk are excluded from this relief.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Evaluate and mitigate systemic risks Art. 55 · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

All 62 obligations by role, risk class and deadline in the obligations compass

More entries from the topic group Model and Training Data Manipulation.

Training Data Poisoning Attackers inject manipulated or false data into the data an AI model learns from. Targeted Poisoning / Label Poisoning Attackers tamper with a small, targeted portion of the training data, for example through wrong labels. Sleepy Agent (Time/Event-Triggered Hidden Instructions) Malicious logic lies dormant inside the model and only activates later: on a set date, at a specific event, or in a particular environment.

Matching use cases

This list shows where to review this threat, rather than how high the risk is.

Typical for:

For 8 further use cases, relevance depends on the system design, for example whether it is trained on your own data or uses another provider's model. Show these use cases

Frequently asked questions

What is Backdoor ML Model?
The model contains hidden behavior, a backdoor. It works correctly on normal inputs; only a secret trigger pattern in the input flips the output to whatever result the attacker has chosen.
Which mitigations help against Backdoor ML Model?
The catalogue lists 4 mitigations, each with control type, effect and implementation level. First among them are Model scanning for backdoors, Neural cleanse techniques and Activation clustering analysis.
Which EU AI Act obligations does Backdoor ML Model touch?
The catalogue maps this threat to Art. 9, Art. 53 Abs. 1 Buchst. a, Art. 53 Abs. 1 Buchst. b, Art. 53 Abs. 1 Buchst. c und d and Art. 55 of the EU AI Act. In the obligations compass this leads to: Establish and operate a risk management system, Draw up the technical documentation of the model, Provide information to downstream providers, Respect copyright and summarise the training content and Evaluate and mitigate systemic risks. Whether an obligation applies depends on role, risk class and deadline.

Assess this threat in your own system

The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.

Start the live demo See the free modules

Cite this entry

For reports, policies or internal documents; the link leads directly to this entry.

“Backdoor ML Model”. Versatile AI Risk Assessment, AI threat catalogue, as of October 2026.
https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/backdoor-ml-model/

← Back to the full catalogue