AI threat catalogueApplication and Integration SecurityProduction
Insecure Tool Design
Many AI applications act on their own through connected tools, for example sending emails or querying databases. If those tools are too powerful or poorly secured, attackers can use the model to trigger actions that were never intended.
Description
Tools are functions or plugins an AI model is allowed to call to get work done. The weakness arises at design time: a tool receives more privileges than it needs (excessive privilege), does not validate the inputs handed to it strictly enough, or exposes unsafe interfaces such as direct access to a command line. Attackers exploit this by steering the model into tool calls with manipulated inputs, for example a prompt injection (instructions smuggled into text the model processes). The tool carries out the action because it trusts the model. Since tools often hold access rights that users are denied directly, they hand attackers additional privileges.
Possible impact
Attackers trigger unauthorised actions with immediate effect: they read, alter or delete data, send emails, initiate bookings or execute code. A single manipulated input thus reaches into connected business systems. This affects operations, data protection and the organisation's liability, including GDPR duties where personal data is involved.
Example
An email assistant is only meant to summarise inboxes, but its tool can also send messages. A crafted incoming email instructs the model to forward confidential correspondence; the tool sends it to the external address.
Recommended mitigations (5)
Every mitigation states its control type, effect, implementation level and the reason for the classification.
Principle of least privilege for toolsTechnical
- Effect
- Preventive, Impact-limiting
- Implementation level
- Application, API & agents
- Complementary control type
- Governance & compliance
- Reason for the classification
- “Principle of least privilege for tools” is primarily technical: Machine-enforced identity, permission, or scope rules constrain unauthorized access and actions; complemented by rules and oversight.
Strict input validation in tool implementationsTechnical
- Effect
- Preventive
- Implementation level
- Application, API & agents
- Reason for the classification
- “Strict input validation in tool implementations” is primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.
Sandboxing and isolationTechnical
- Effect
- Preventive, Impact-limiting
- Implementation level
- Application, API & agents, Infrastructure
- Reason for the classification
- “Sandboxing and isolation” is primarily technical: Architectural and runtime boundaries separate data, execution, or tenants and limit the spread of failures.
Human-in-the-loop for sensitive operationsOrganizational & process-based
- Effect
- Preventive
- Implementation level
- Application, API & agents, Use & operations
- Complementary control type
- People & competence
- Reason for the classification
- “Human-in-the-loop for sensitive operations” is primarily organizational and process-based: A binding workflow requires an accountable human decision before use or execution; complemented by human expertise and judgment.
Comprehensive tool security reviewOrganizational & process-based
- Effect
- Detective
- Implementation level
- Application, API & agents, Organization, Use & operations
- Complementary control type
- Governance & compliance
- Reason for the classification
- “Comprehensive tool security review” is primarily organizational and process-based: A planned, repeatable assessment with ownership and documented follow-up creates the protective effect; complemented by rules and oversight.
Framework mappings
Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.
Verified references (8)
Every reference states the framework, the exact location and the publishing organisation.
- OWASP LLM Top 10 ASI02:2026 Tool Misuse and ExploitationASI02 Tool Misuse and Exploitation, pp. 12–14 of the official PDF OWASP FoundationOriginal
- OWASP LLM Top 10 LLM06:2025 Excessive AgencyLLM06:2025 Excessive Agency, official category page OWASP FoundationOriginal
- NIST AI RMF Section 2.9 Information SecuritySection 2.9, pp. 10–11 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF Section 3.5 Security of AgentsSection 3.5, p. 54 National Institute of Standards and Technology (NIST)Original
- MITRE ATLAS AML.T0053 AI Agent Tool InvocationATLAS.yaml technique object with id AML.T0053 (pinned release v5.6.0) MITREOriginal
- EU AI Act Article 14(4)(d) Human oversightArticle 14(4)(d); for automation bias, Article 14(4)(b) European Union (EUR-Lex)Original
- EU AI Act Article 9(1), 9(2)(a), 9(2)(d) Risk management systemArticle 9(1), 9(2)(a), 9(2)(d), read with Article 9(3) European Union (EUR-Lex)Original
- BSI R28 Indirect Prompt Injections (Text)Kap. 4, R28, p. 33 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
Terms on this page
Glossary terms that occur in this entry. Every link leads to the full explanation.
- Prompt Injection Manipulated input or planted content redirects a language model.
Related threats
More entries from the topic group Application and Integration Security.
Assess this threat in your own system
The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.
Cite this entry
For reports, policies or internal documents; the link leads directly to this entry.
“Insecure Tool Design”. Versatile AI Risk Assessment, AI threat catalogue, as of July 2026. https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/insecure-tool-design/