Versatile AI Risk Assessment

AI threat catalogue Application and Integration Security Production

Insecure Tool Design

Many AI applications act on their own through connected tools, for example sending emails or querying databases. If those tools are too powerful or poorly secured, attackers can use the model to trigger actions that were never intended.

As of: October 2026 · Catalogue version 2026.10.24.2 · 5 mitigations · 9 verified sources

Description

Tools are functions or plugins an AI model is allowed to call to get work done. The weakness arises at design time: a tool receives more privileges than it needs (excessive privilege), does not validate the inputs handed to it strictly enough, or exposes unsafe interfaces such as direct access to a command line. Attackers exploit this by steering the model into tool calls with manipulated inputs, for example a prompt injection (instructions smuggled into text the model processes). The tool carries out the action because it trusts the model. Since tools often hold access rights that users are denied directly, they hand attackers additional privileges.

Possible impact

Attackers trigger unauthorised actions with immediate effect: they read, alter or delete data, send emails, initiate bookings or execute code. A single manipulated input thus reaches into connected business systems. This affects operations, data protection and the organisation's liability, including GDPR duties where personal data is involved.

Example

An email assistant is only meant to summarise inboxes, but its tool can also send messages. A crafted incoming email instructs the model to forward confidential correspondence; the tool sends it to the external address.

Recommended mitigations (5)

Compare effects and implementation levels. Expand each entry for the additional classification and rationale.

  • Principle of least privilege for tools

    Technical
    Effect
    Preventive, Impact-limiting
    Implementation level
    Application, API & agents
    Classification and rationale

    Complementary control typeGovernance & compliance

    Primarily technical: Machine-enforced identity, permission, or scope rules constrain unauthorized access and actions; complemented by rules and oversight.

  • Strict input validation in tool implementations

    Technical
    Effect
    Preventive
    Implementation level
    Application, API & agents
    Classification and rationale

    Primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.

  • Sandboxing and isolation

    Technical
    Effect
    Preventive, Impact-limiting
    Implementation level
    Application, API & agents, Infrastructure
    Classification and rationale

    Primarily technical: Architectural and runtime boundaries separate data, execution, or tenants and limit the spread of failures.

  • Human-in-the-loop for sensitive operations

    Organizational & process-based
    Effect
    Preventive
    Implementation level
    Application, API & agents, Use & operations
    Classification and rationale

    Complementary control typePeople & competence

    Primarily organizational and process-based: A binding workflow requires an accountable human decision before use or execution; complemented by human expertise and judgment.

  • Comprehensive tool security review

    Organizational & process-based
    Effect
    Detective
    Implementation level
    Application, API & agents, Organization, Use & operations
    Classification and rationale

    Complementary control typeGovernance & compliance

    Primarily organizational and process-based: A planned, repeatable assessment with ownership and documented follow-up creates the protective effect; complemented by rules and oversight.

Framework mappings

Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.

OWASP LLM Top 10 ASI02:2026 · LLM03:2026 · LLM10:2026NIST AI RMF Section 2.9 · Section 3.5MITRE ATLAS AML.T0053EU AI Act Article 14(4)(d) · Article 9(1), 9(2)(a), 9(2)(d)BSI R28

Verified references (9)

Every reference states the framework, the exact location and the publishing organisation.

EU AI Act obligations related to this threat

The obligations from the obligations compass whose articles the catalogue maps to this threat. Each leads to its entry with core, action step and evidence; whether it applies to you depends on role and risk class.

  • Establish and operate a risk management system Art. 9 · applies from Annex III: 2 December 2027 · Annex I: 2 August 2028

    Statutory roles: Provider, Product manufacturer. Classes: High-risk under Annex III, High-risk as a product or safety component.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Build in effective human oversight Art. 14 · applies from Annex III: 2 December 2027 · Annex I: 2 August 2028

    Statutory roles: Provider, Product manufacturer. Classes: High-risk under Annex III, High-risk as a product or safety component.

    This reference is a review prompt. A threat match alone does not establish applicability.

All 62 obligations by role, risk class and deadline in the obligations compass

Terms on this page

Glossary terms that occur in this entry. Every link leads to the full explanation.

  • Prompt Injection Manipulated input or planted content redirects a language model.
  • Least privilege Gives people, services and agents only the permissions their task requires.

More entries from the topic group Application and Integration Security.

Insecure Output Handling The application passes an AI model's answers to downstream systems without checks, for example to databases, web pages or command lines. Application Vulnerabilities The software around the AI model, meaning its web interface, APIs and databases, has the same weaknesses as any other application. RAG-Specific Attacks (Document Poisoning) Attackers plant prepared documents in the knowledge base an AI system draws on via RAG (retrieval-augmented generation, looking up …

Matching use cases

This list shows where to review this threat, rather than how high the risk is.

Particularly important for:

  • AI agentAn agent's tool accepts arbitrary inputs without checking who initiated the call. The agent thus reaches data and functions unrelated to its task.
  • Coding agentUnrestricted terminal access lets the agent execute any command available to the signed-in user. Without isolation and an allowlist of commands, a mistake is difficult to contain.

For 7 further use cases, relevance depends on the system design, for example whether it calls other programs or interfaces or is connected through the Model Context Protocol (MCP). Show these use cases

Frequently asked questions

What is Insecure Tool Design?
Many AI applications act on their own through connected tools, for example sending emails or querying databases. If those tools are too powerful or poorly secured, attackers can use the model to trigger actions that were never intended.
Which mitigations help against Insecure Tool Design?
The catalogue lists 5 mitigations, each with control type, effect and implementation level. First among them are Principle of least privilege for tools, Strict input validation in tool implementations and Sandboxing and isolation.
Which EU AI Act obligations does Insecure Tool Design touch?
The catalogue maps this threat to Art. 9 and Art. 14 of the EU AI Act. In the obligations compass this leads to: Establish and operate a risk management system and Build in effective human oversight. Whether an obligation applies depends on role, risk class and deadline.

Assess this threat in your own system

The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.

Start the live demo See the free modules

Cite this entry

For reports, policies or internal documents; the link leads directly to this entry.

“Insecure Tool Design”. Versatile AI Risk Assessment, AI threat catalogue, as of October 2026.
https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/insecure-tool-design/

← Back to the full catalogue