Versatile AI Risk Assessment

AI threat catalogueAttacks on the Running Model and ServiceProduction

Adversarial Inputs

Attackers alter an input in a way that is minimal and imperceptible to humans, so that the AI model classifies it incorrectly. To the system a stop sign becomes something else, even though a person sees no difference.

As of: July 2026 · Catalogue version 2026.07.17.3 · 4 mitigations · 14 verified sources

Description

Adversarial inputs are specially crafted data that push the model to a wrong output while looking normal to humans. Attackers compute the necessary, usually tiny change with knowledge of the model (gradient methods) or probe it through many test queries; sometimes they practise on a rebuilt substitute model and transfer the attack. For text, a character swap (such as "$" for "S"), rare synonyms or interspersed character sequences often suffice. The attack happens at run time, not during training. This makes it possible to bypass AI-based protective functions such as malware and spam detection, hate-speech filters or biometric checks.

Possible impact

Important decisions come out wrong: malware judged safe passes detection, a prohibited input slips through the filter, a person is matched incorrectly by biometrics. In safety-critical applications such as image or traffic-sign recognition, this can directly endanger people. The EU AI Act explicitly requires high-risk systems to include measures against inputs designed to cause the model to make mistakes (adversarial examples, model evasion).

Example

An AI filter is meant to block offensive comments. An attacker replaces individual letters with similar-looking special characters and inserts meaningless character strings; the text stays readable for humans, but the filter no longer recognises the violation.

Recommended mitigations (4)

Every mitigation states its control type, effect, implementation level and the reason for the classification.

Framework mappings

Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.

OWASP LLM Top 10 LLM09:2025NIST AI RMF Section 2.9 · MEASURE 2.7 · NISTAML.022 · NISTAML.025MITRE ATLAS AML.T0015EU AI Act Article 26(5) · Article 55(1)(a) · Article 9(1), 9(2)(a), 9(2)(d)BSI R27 · R3BIML BIML78 inference:3 · BIML78 input:1 · BIML78 input:2

Verified references (14)

Every reference states the framework, the exact location and the publishing organisation.

More entries from the topic group Attacks on the Running Model and Service.

Model Theft Attackers create a working copy of a proprietary AI model, either by querying it at scale or by breaking into the infrastructure. They … Model Denial of Service Attackers overload the AI model with especially expensive requests until the service slows down or fails. They exploit the fact that AI … Cost Harvesting / Repurposing Attackers use a company's paid AI services without authorisation and at its expense, for example with stolen access keys. The costs and the …

Assess this threat in your own system

The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.

Start the live demo See the free modules

Cite this entry

For reports, policies or internal documents; the link leads directly to this entry.

“Adversarial Inputs”. Versatile AI Risk Assessment, AI threat catalogue, as of July 2026.
https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/adversarial-inputs/

← Back to the full catalogue