Input data and information sources
Tickets, external code, dependencies and websites the agent reads can come from outsiders. Code repositories, test data and logs can contain personal information.
Coding agent
Also known as: AI coding agent
The agent takes on complete programming tasks, works in the code repository and terminal, executes commands itself and submits changes for review.
Understand the use case
What defines this use case?
Goldman Sachs is trialling such an agent; according to the manufacturer, Nubank used one to complete a major code migration.
This page describes a typical system design. Actual data flows and permissions depend on your implementation.
- System design
- Agent with tool access
- Ability to act
- Can act within applications
- Fields of use
- financial services
- Also known as
- AI coding agent, programming agent, software agent
How it works, step by step
System design and data flows
Input data
The task arrives as a ticket or message; the agent also reads the repository, documentation and, if needed, websites.
Tickets, external code, dependencies and websites the agent reads can come from outsiders. Code repositories, test data and logs can contain personal information.
Processing
A language model plans steps, edits files, runs commands and tests, and adapts its approach to the results.
The model plans project changes, reads relevant files and calls approved development tools. Results from tests or other calls provide context for the next step. A successfully executed tool action must be distinguished from a substantively correct and approved change.
Results
The completed changes are provided as a branch or pull request with a description.
A generated result is newly worded or created. To assess it, the instruction and context on which it is based must remain identifiable.
Use and impact
Commands affect the development environment while work is underway. After review and merge, the change enters the shipped software.
Write or execution permissions allow the system to change a state within an application. Model responses, tool calls and changes actually executed must therefore be considered separately.
Explore the system design: Select a component to see its task, incoming and outgoing data, and role in the use case. The guided walkthrough explains each phase. Then use Yes, Uncertain and No to represent your specific system variant.
16 review first25 clarify9 baseline threats
- Present
- Depends on your system
- Absent
- Review first
- To clarify
How to use the model
The diagram shows the typical architecture. Select a component to see what happens there and adapt its features to your system. Enable “Analysis guidance” to see which threats need reviewing for this architecture. The page's checklist continues to describe the typical case.
Changes from the typical case
None yet. Each change will appear here.
Where most threats arise
- 3 review first, 12 clarify
- 6 review first
- 3 review first, 2 clarify
- 4 clarify
In this system design: 16 to review first, 25 to clarify, 9 baseline threats.
Architecture as a list
External access: Depends on your system
To clarify
Input: Part of every AI system
Review first
To clarify
Baseline for every AI system
Third-party content: Present
Review first
- Prompt Injection: Indirect Review priority
AI model: Part of every AI system
Review first
- Sleepy Agent (Time/Event-Triggered Hidden Instructions)
- Prompt Injection: Direct
- Jailbreaks
- Meta Prompt Extraction
- Misalignment
- Agentic AI / Autonomous Agents
Baseline for every AI system
Provider service: Present
Review first
To clarify
Document collection: Depends on your system
To clarify
Memory: Depends on your system
To clarify
Training and feedback: Depends on your system
To clarify
Other AI agents: Depends on your system
Tools and interfaces: Present
Review first
- Exfiltration from ML Application Review priority
- Insecure Tool Design Review priority
- Excessive Agency Review priority
To clarify
Result: Part of every AI system
Review first
To clarify
- Hate Speech and Discrimination
- Profanity
- Sexual Content
- Violence / Unsafe Actions
- Controversial Topics
- Illegal Activities
- Self-harm
- Harassment
- Unethical Actions
- Social Engineering
- Fraud
- Disinformation
Baseline for every AI system
Business application or equipment: Present
person: Part of every AI system
Baseline for every AI system
Unapproved AI services: Depends on your system
To clarify
Supply chain and technical foundations: 5 baseline threats applying to every AI system
Baseline for every AI system
How it differs from similar use cases
Only suggests code for developers to review and accept themselves
Completes tasks in office applications and business systems rather than source code
Data and information
Processing and operating environment
Agents use a provider's language model.
Storage and further use
Many agents search code repositories and documentation for relevant material. Some agents store notes about projects and earlier tasks and consult them later.
Results and data sharing
The agent calls the terminal, version control, package managers and testing tools. The agent edits files, creates branches and opens pull requests.
Using a provider does not automatically determine where processing takes place. Storage location, subcontractors, deletion periods and use for training must be checked for the specific deployment.
Assess risks and determine measures
From understanding the system to analysing risks
- Define the system boundary
Record data, participants, interfaces and permissions. Resolve uncertain characteristics in the model.
- Investigate threats
Relate review candidates to actual data flows: what could happen, and who would be affected?
- Assess risk
Assess impacts and likelihood in the deployment context. The number of review candidates is not a risk score.
- Provide evidence of measures
Select appropriate controls, assign responsibilities and define effectiveness checks. Document residual risks and approvals.
Key threats in the application context
- 4 particularly important
- 12 typical
- 25 depending on the system
- 9 for every AI system
A total of 50 threats from the threat catalogue may be relevant to this use case.
Priority review points
A ticket, README file or website contains hidden instructions. The agent executes them with its permissions, for example deleting files or reading credentials.
Permission to write to the main branch, production access credentials or cloud accounts lets mistakes cause damage beyond development. An unreviewed merge can send a change straight into production.
Unrestricted terminal access lets the agent execute any command available to the signed-in user. Without isolation and an allowlist of commands, a mistake is difficult to contain.
Exfiltration from ML Application
The agent can open network connections and execute commands. An injected instruction can make it send source code or secrets to an external address.
12 further review candidates in the typical design
Other relevant threats
Training data or an adopted base model can introduce hidden triggers for misbehaviour.
Listed because of this characteristic
- Uses another provider's model
Sleepy Agent (Time/Event-Triggered Hidden Instructions)
An instruction-following model can change behaviour at a particular time or event.
Listed because of this characteristic
- Performs natural language tasks
Users can override the system's working instructions through their inputs.
Listed because of this characteristic
- Performs natural language tasks
People requesting generated content may try to bypass intended limits.
Listed because of this characteristic
- Performs natural language tasks
Carefully crafted queries can reveal system instructions, which must therefore contain no secrets.
Listed because of this characteristic
- Performs natural language tasks
Complex or high-volume inputs can exhaust model computing resources.
Listed because of this characteristic
- Generates its own text
Targeted queries can recover some personal or protected training data from the model.
Listed because of this characteristic
- Processes confidential data
Sensitive Information Disclosure
Confidential content can appear in answers, supplied context or logs.
Listed because of this characteristic
- Processes confidential data
Generated code can contain malicious functions or help develop them.
Listed because of this characteristic
- Generates code or queries
Factual Inconsistencies (Hallucinations)
Generated text can invent facts or distort them during rewriting.
Listed because of this characteristic
- Generates its own text
The target metric can improve while the underlying objective is missed.
Listed because of this characteristic
- Determines its own work steps
Agentic AI / Autonomous Agents
When a system plans and acts over several steps, an error can compound at each step.
Listed because of this characteristic
- Determines its own work steps
Up to 25 further threats may apply, depending on your system design. Nine threats need to be reviewed for every AI system.
System features and open questions
Each Yes adds further threats to your checklist, up to 25 in total. If you cannot yet answer a question, include its threats in your review.
- Added by this Yes
- Also requires a Yes to another question
How your system is built
- Generates content from open-ended requests10 threats, plus 2 only in combination with other characteristics
- Uses the Model Context Protocol (MCP)2 threats
- Uses operational feedback2 threats
- Learns from your own data1 threat, plus 1 only in combination with other characteristics
- Searches its own document collection1 threat
- Formulates answers from retrieved documents1 threat
- Remembers content beyond a single conversation1 threat
- Works with other AI agents1 threat
- Learns from examples with correct answers1 threat
- Processes text, images and speech together1 threat
How you operate the system
- Available to people outside the organisation4 threats, plus 2 only in combination with other characteristics
- Accepts queries directly through an interface4 threats
- Accessible over a network1 threat, plus 1 only in combination with other characteristics
- Incurs costs with each query1 threat
- Shares computing resources with other customers or applications1 threat
- Used alongside unapproved AI services1 threat
How your system is built
Does the system create new content from open-ended requests that people read, see or hear directly? Translation, reading aloud and transcription do not count.
Depending on the setup, the agent also answers open-ended questions beyond coding.
If yes, add
Does the system search its own collection of documents or records for relevant content?
Many agents search code repositories and documentation for relevant material.
If yes, add
Does the system formulate answers from retrieved documents?
If yes, add
Does the system remember content from earlier conversations or tasks and reuse it later?
Some agents store notes about projects and earlier tasks and consult them later.
If yes, add
Does the system delegate tasks to other AI agents or incorporate their results?
Some agents delegate subtasks to other agents and incorporate their results.
If yes, add
Is the system connected to tools or data sources through the Model Context Protocol (MCP)?
Agents are often connected to additional tools and data sources through the Model Context Protocol (MCP).
If yes, add
Does your organisation train or fine-tune the model itself using its own data?
If yes, add
Is the model trained on examples to which someone has assigned the correct answer?
If yes, add
Does operational feedback or output later feed into analysis or training?
Accepted and rejected changes may feed into the provider's further development.
If yes, add
Does the system process several input types together, such as text with images or speech?
If yes, add
How you operate the system
These questions arise for almost every AI system.
Can people or programs query the model directly, for example through an interface?
If yes, add
Is the system accessible over a network or dependent on network services?
If yes, add
Do you pay per query or unit of computing time, so costs increase with use?
If yes, add
Does the system share servers or computing resources with other customers or applications?
If yes, add
Can people outside your organisation use the system, including without login?
If yes, add
Do staff also use AI services for this task that your organisation has not approved?
If yes, add
Controls and implementation
Measures for the priority review areas are expanded. Each linked measure explains how to implement it.
Review priorityPrompt Injection: Indirect3 measures
Review priorityExcessive Agency3 measures
Review priorityInsecure Tool Design3 measures
Review priorityExfiltration from ML Application3 measures
Backdoor ML Model2 measures
Sleepy Agent (Time/Event-Triggered Hidden Instructions)3 measures
Prompt Injection: Direct3 measures
Jailbreaks2 measures
Meta Prompt Extraction2 measures
Model Denial of Service3 measures
Privacy Attacks3 measures
Sensitive Information Disclosure3 measures
Malicious Software3 measures
Factual Inconsistencies (Hallucinations)3 measures
Misalignment3 measures
Agentic AI / Autonomous Agents3 measures
Controls for the remaining threats are available on their pages in the threat catalogue.
AI Act provisions
-
Art. 4: AI literacy
For every AI system Applies since 2 February 2025.
-
Art. 53 para. 1 point b, Annex XII: Embedded general-purpose model
Depending on use Review trigger since 2 August 2025.
Applies to every AI system
Art. 4: AI literacy
Applies if staff or appointed representatives operate or use AI systems.
Applies since 2 February 2025.
Obligations and details for Art. 4: AI literacy
- Who bears the obligations
- Provider, Deployer, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
May apply in particular uses
Art. 53 para. 1 point b, Annex XII: Embedded general-purpose model
Applies if a large language model or another third-party general-purpose model is incorporated into an agent you provide yourself.
Applies since 2 August 2025.
Obligations and details for Art. 53 para. 1 point b, Annex XII: Embedded general-purpose model
- The general condition is
- Applies if another provider's general-purpose AI model is incorporated, such as a large language model.
- Points to consider
- The model provider bears its obligations. It must give you the Annex XII information; use it for your own documentation.
- Who bears the obligations
- Provider of a general-purpose AI model
Detailed obligations
Classify your system with the quick checker All obligations in the obligations compass
Cross-cutting threats
Review these threats for every AI system, whatever its purpose.
Atypical system features
View 2 further questions
If any of these features does apply to your system, add the listed threats.
Does the system use a knowledge graph storing concepts and their relationships?
If yes, add
Do documents from several customers, departments or permission groups share one search store?
If yes, add
Sources
- Report of a deployment
CNBC, Goldman Sachs is piloting its first autonomous coder in major AI milestone for Wall Street, 11 July 2025 According to the deployer
The independent report describes a bank's trial of a coding agent.
- Manufacturer
Cognition, Nubank customer case study According to the manufacturer
The manufacturer describes the agent migrating data classes from a large ETL system at Nubank.
- Register or case collection
MITRE ATLAS, atlas-data CHANGELOG
Case studies AML.CS0047, AML.CS0067 and AML.CS0069 describe malicious instructions in Amazon Q for VS Code, exposed secrets and an espionage campaign using Claude Code.
- Legislation, norm or standard
OWASP GenAI Security Project, OWASP Top 10 for Agentic Applications 2026
The list identifies unexpected code execution (ASI05) and tool misuse (ASI02) as distinct risks in agentic applications.
- Manufacturer
Anthropic, Disrupting the first reported AI-orchestrated cyber espionage campaign, November 2025 According to the manufacturer
The model provider reports that attackers abused its Claude Code programming agent to carry out largely autonomous attacks against around 30 targets.
Related use cases
Also in the field of Software development and IT security
- Coding assistantThe assistant suggests code, tests and fixes in the development environment for developers to review and accept themselves.
- AI-based cyberattack detectionThe system learns normal patterns in networks, devices, accounts and email traffic, then flags deviations that may indicate attacks, phishing or compromised accounts.
Updated: 30 September 2026. Review checklist rules version 0.3.2, threat catalogue version 2026.10.24.2. How these pages are structured