Versatile AI Risk Assessment

AI threat catalogue Application and Integration Security Production

Insecure Output Handling

The application passes an AI model's answers to downstream systems without checks, for example to databases, web pages or command lines. If the output contains malicious code or commands, the system executes them as if they came from a trusted source.

As of: October 2026 · Catalogue version 2026.10.24.2 · 4 mitigations · 9 verified sources

Description

Many AI applications do more than display model answers: they render them as web pages, build database queries from them or run generated program code. Because attackers can steer what the model outputs through their inputs, this gives them indirect access to those downstream systems. Without validation and context-aware encoding of the outputs, classic injection attacks emerge, in which attackers smuggle their own commands into a system: malicious scripts in the browser (XSS), manipulated database commands (SQL injection), and even attacker-supplied code running on the server (remote code execution). The risk grows when the AI application holds more privileges than the person operating it.

Possible impact

A successful attack gives attackers control over parts of the application: they read or alter data, hijack other users' sessions or run their own commands on the servers. For the organisation this means operational disruption, data leakage with notification duties under the GDPR and a loss of customer trust, even though the model itself worked flawlessly.

Example

A maths assistant runs model-generated program code directly on the server to solve calculation tasks. A user phrases the question so that the model produces malicious code; the server executes it and gives the attacker access to the system.

Recommended mitigations (4)

Compare effects and implementation levels. Expand each entry for the additional classification and rationale.

  • Treat model output as untrusted

    Organizational & process-based
    Effect
    Preventive
    Implementation level
    Application, API & agents, Organization
    Classification and rationale

    Complementary control typeTechnical

    Primarily organizational and process-based: Defined selection, operating, or lifecycle procedures make the control binding and repeatable; complemented by technical implementation.

  • Output validation and encoding

    Technical
    Effect
    Preventive
    Implementation level
    Application, API & agents
    Classification and rationale

    Primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.

  • Context-aware sanitization

    Technical
    Effect
    Preventive
    Implementation level
    Application, API & agents
    Classification and rationale

    Primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.

  • Avoid direct execution of model output

    Technical
    Effect
    Preventive
    Implementation level
    Application, API & agents
    Classification and rationale

    Primarily technical: Safe formats, restrictive defaults, or protective layers reduce unsafe execution paths and exploitable attack surface.

Framework mappings

Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.

OWASP LLM Top 10 LLM08:2026 · LLM10:2026NIST AI RMF Section 2.9MITRE ATLAS AML.T0048EU AI Act Article 9(1), 9(2)(a), 9(2)(d)BSI R16 · R6BIML BIML-LLM inference:8 · BIML78 system:9

Verified references (9)

Every reference states the framework, the exact location and the publishing organisation.

EU AI Act obligations related to this threat

The obligations from the obligations compass whose articles the catalogue maps to this threat. Each leads to its entry with core, action step and evidence; whether it applies to you depends on role and risk class.

  • Establish and operate a risk management system Art. 9 · applies from Annex III: 2 December 2027 · Annex I: 2 August 2028

    Statutory roles: Provider, Product manufacturer. Classes: High-risk under Annex III, High-risk as a product or safety component.

    This reference is a review prompt. A threat match alone does not establish applicability.

All 62 obligations by role, risk class and deadline in the obligations compass

More entries from the topic group Application and Integration Security.

Insecure Tool Design Many AI applications act on their own through connected tools, for example sending emails or querying databases. Application Vulnerabilities The software around the AI model, meaning its web interface, APIs and databases, has the same weaknesses as any other application. RAG-Specific Attacks (Document Poisoning) Attackers plant prepared documents in the knowledge base an AI system draws on via RAG (retrieval-augmented generation, looking up …

Matching use cases

This list shows where to review this threat, rather than how high the risk is.

Belongs on the checklist for every AI system. What to review for every AI system

Particularly important for:

  • Machine translationAn automatically translated website goes live without review. Errors in deadlines, fees or responsibilities reach citizens or customers directly.
  • Intelligent document processing (IDP)Extracted fields go straight into the business system. Accepting them unchecked lets incorrect amounts or references trigger payments or deadlines.
  • Customer service chatbots and voicebotsThe bot's answer reaches the customer directly without review. If it gives an incorrect deadline or refund rule, the customer acts on it and the deployer must answer for it, as in the Air Canada case.
  • Dynamic pricingCalculated prices go straight into the shop without review. An input error then creates prices far below cost or far above market levels that take effect immediately.
  • AI-assisted insurance claims handlingThe proposal passes directly into the claims system with amount and coverage assessment. Accepting an incorrect amount unchecked leads to overpayment or underpayment.
  • AI-assisted candidate screeningThe ranking passes into the applicant tracking system and can trigger rejections. A scoring error thus becomes a direct decision about a person.
  • Biometric verification and video identificationDownstream processes treat the result as a verified identity. Accepting it without further checks means a single error can create an account or grant access under a false name.
  • Risk-based review selection for applications and benefitsSelection passes into the business system and can stop payments. An incorrect risk score delays benefits for someone who depends on them.
  • AI assistance in high-volume court proceedingsExtracted features such as vehicle type, purchase date or flight delay select the draft's standard text passages. A misread feature inserts the wrong legal reasoning.
  • AI-assisted medical image analysisIf the system prioritises its worklist by classification, apparently normal images move down the queue. A misclassification delays review of an urgent finding.

Frequently asked questions

What is Insecure Output Handling?
The application passes an AI model's answers to downstream systems without checks, for example to databases, web pages or command lines. If the output contains malicious code or commands, the system executes them as if they came from a trusted source.
Which mitigations help against Insecure Output Handling?
The catalogue lists 4 mitigations, each with control type, effect and implementation level. First among them are Treat model output as untrusted, Output validation and encoding and Context-aware sanitization.
Which EU AI Act obligations does Insecure Output Handling touch?
The catalogue maps this threat to Art. 9 of the EU AI Act. In the obligations compass this leads to: Establish and operate a risk management system. Whether an obligation applies depends on role, risk class and deadline.

Assess this threat in your own system

The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.

Start the live demo See the free modules

Cite this entry

For reports, policies or internal documents; the link leads directly to this entry.

“Insecure Output Handling”. Versatile AI Risk Assessment, AI threat catalogue, as of October 2026.
https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/insecure-output-handling/

← Back to the full catalogue