Versatile AI Risk Assessment

AI threat catalogue Attacks on the Running Model and Service Production

Adversarial Inputs

Attackers alter an input in a way that is minimal and imperceptible to humans, so that the AI model classifies it incorrectly. To the system a stop sign becomes something else, even though a person sees no difference.

As of: October 2026 · Catalogue version 2026.10.24.2 · 4 mitigations · 14 verified sources

Description

Adversarial inputs are specially crafted data that push the model to a wrong output while looking normal to humans. Attackers compute the necessary, usually tiny change with knowledge of the model (gradient methods) or probe it through many test queries; sometimes they practise on a rebuilt substitute model and transfer the attack. For text, a character swap (such as "$" for "S"), rare synonyms or interspersed character sequences often suffice. The attack happens at run time, not during training. This makes it possible to bypass AI-based protective functions such as malware and spam detection, hate-speech filters or biometric checks.

Possible impact

Important decisions come out wrong: malware judged safe passes detection, a prohibited input slips through the filter, a person is matched incorrectly by biometrics. In safety-critical applications such as image or traffic-sign recognition, this can directly endanger people. The EU AI Act explicitly requires high-risk systems to include measures against inputs designed to cause the model to make mistakes (adversarial examples, model evasion).

Example

An AI filter is meant to block offensive comments. An attacker replaces individual letters with similar-looking special characters and inserts meaningless character strings; the text stays readable for humans, but the filter no longer recognises the violation.

Recommended mitigations (4)

Compare effects and implementation levels. Expand each entry for the additional classification and rationale.

  • Adversarial training

    Technical
    Effect
    Preventive
    Implementation level
    Model & training
    Classification and rationale

    Primarily technical: A model, training, or data-processing method directly changes system behavior or robustness.

  • Input validation and preprocessing

    Technical
    Effect
    Preventive
    Implementation level
    Model & training, Application, API & agents
    Classification and rationale

    Primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.

  • Certified or randomised robustness methods and robustness testing

    Technical
    Effect
    Preventive
    Implementation level
    Model & training
    Classification and rationale

    Primarily technical: A model, training, or data-processing method directly changes system behavior or robustness.

  • Ensemble models

    Technical
    Effect
    Preventive
    Implementation level
    Model & training
    Classification and rationale

    Primarily technical: A model, training, or data-processing method directly changes system behavior or robustness.

Framework mappings

Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.

OWASP LLM Top 10 LLM07:2026NIST AI RMF Section 2.9 · MEASURE 2.7 · NISTAML.022 · NISTAML.025MITRE ATLAS AML.T0015EU AI Act Article 26(5) · Article 55(1)(a) · Article 9(1), 9(2)(a), 9(2)(d)BSI R27 · R3BIML BIML78 inference:3 · BIML78 input:1 · BIML78 input:2

Verified references (14)

Every reference states the framework, the exact location and the publishing organisation.

EU AI Act obligations related to this threat

The obligations from the obligations compass whose articles the catalogue maps to this threat. Each leads to its entry with core, action step and evidence; whether it applies to you depends on role and risk class.

  • Establish and operate a risk management system Art. 9 · applies from Annex III: 2 December 2027 · Annex I: 2 August 2028

    Statutory roles: Provider, Product manufacturer. Classes: High-risk under Annex III, High-risk as a product or safety component.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Monitor operation and suspend on risk Art. 26 Abs. 5 · applies from Annex III: 2 December 2027 · Annex I: 2 August 2028

    Statutory roles: Deployer. Classes: High-risk under Annex III, High-risk as a product or safety component.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Evaluate and mitigate systemic risks Art. 55 · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

All 62 obligations by role, risk class and deadline in the obligations compass

More entries from the topic group Attacks on the Running Model and Service.

Model Theft Attackers create a working copy of a proprietary AI model, either by querying it at scale or by breaking into the infrastructure. Model Denial of Service Attackers overload the AI model with especially expensive requests until the service slows down or fails. Cost Harvesting / Repurposing Attackers use a company's paid AI services without authorisation and at its expense, for example with stolen access keys.

Matching use cases

This list shows where to review this threat, rather than how high the risk is.

Belongs on the checklist for every AI system. What to review for every AI system

Particularly important for:

Frequently asked questions

What is Adversarial Inputs?
Attackers alter an input in a way that is minimal and imperceptible to humans, so that the AI model classifies it incorrectly. To the system a stop sign becomes something else, even though a person sees no difference.
Which mitigations help against Adversarial Inputs?
The catalogue lists 4 mitigations, each with control type, effect and implementation level. First among them are Adversarial training, Input validation and preprocessing and Certified or randomised robustness methods and robustness testing.
Which EU AI Act obligations does Adversarial Inputs touch?
The catalogue maps this threat to Art. 9, Art. 26 Abs. 5 and Art. 55 of the EU AI Act. In the obligations compass this leads to: Establish and operate a risk management system, Monitor operation and suspend on risk and Evaluate and mitigate systemic risks. Whether an obligation applies depends on role, risk class and deadline.

Assess this threat in your own system

The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.

Start the live demo See the free modules

Cite this entry

For reports, policies or internal documents; the link leads directly to this entry.

“Adversarial Inputs”. Versatile AI Risk Assessment, AI threat catalogue, as of October 2026.
https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/adversarial-inputs/

← Back to the full catalogue