Input data and information sources
External code, dependencies, tickets and online documentation enter the assistant's context. Source code, test data and logs can contain personal information.
Coding assistant
Also known as: AI coding assistant
The assistant suggests code, tests and fixes in the development environment for developers to review and accept themselves.
Understand the use case
What defines this use case?
A language model reads the open file and additional project context and produces suggestions in the editor or chat. Mercedes-Benz, Germany's Federal Employment Agency and a Dutch archive use such assistants. Security agencies BSI and ANSSI have published a dedicated report.
This page describes a typical system design. Actual data flows and permissions depend on your implementation.
- System design
- Generative assistance
- Ability to act
- Provides results for subsequent use
- Fields of use
- automotive industry, public administration
- Also known as
- AI programming assistant, code completion, GitHub Copilot
How it works, step by step
System design and data flows
Input data
The assistant reads the open file, additional project context and the developer's request.
External code, dependencies, tickets and online documentation enter the assistant's context. Source code, test data and logs can contain personal information.
Processing
A language model uses these to complete code or answer the request.
The language model processes the instruction together with the available code context and produces a suggestion for code, tests or documentation. The development environment presents this suggestion for adoption. Execution, testing and integration into the project involve other tools and the required approvals.
Results
Code, tests, commands or explanations appear as suggestions in the editor.
A generated result is newly worded or created. To assess it, the instruction and context on which it is based must remain identifiable.
Use and impact
Accepted suggestions enter the source code and, with the next release, the software running in production.
The model supplies information for subsequent use. Selection, adoption or a professional decision in the following workflow is a separate step and should not automatically be attributed to the model.
Explore the system design: Select a component to see its task, incoming and outgoing data, and role in the use case. The guided walkthrough explains each phase. Then use Yes, Uncertain and No to represent your specific system variant.
13 review first27 clarify8 baseline threats
- Present
- Depends on your system
- Absent
- Review first
- To clarify
How to use the model
The diagram shows the typical architecture. Select a component to see what happens there and adapt its features to your system. Enable “Analysis guidance” to see which threats need reviewing for this architecture. The page's checklist continues to describe the typical case.
Changes from the typical case
None yet. Each change will appear here.
Where most threats arise
- 3 review first, 12 clarify
- 4 review first
- 3 review first, 1 clarify
- 4 clarify
In this system design: 13 to review first, 27 to clarify, 8 baseline threats.
Architecture as a list
External access: Depends on your system
To clarify
Input: Part of every AI system
Review first
To clarify
Baseline for every AI system
Third-party content: Present
Review first
- Prompt Injection: Indirect Review priority
AI model: Part of every AI system
Review first
- Sleepy Agent (Time/Event-Triggered Hidden Instructions)
- Prompt Injection: Direct
- Jailbreaks
- Meta Prompt Extraction
Baseline for every AI system
Provider service: Present
Review first
To clarify
Document collection: Depends on your system
To clarify
Memory: Depends on your system
To clarify
Training and feedback: Depends on your system
To clarify
Tools and interfaces: Depends on your system
To clarify
Result: Part of every AI system
Review first
To clarify
- Hate Speech and Discrimination
- Profanity
- Sexual Content
- Violence / Unsafe Actions
- Controversial Topics
- Illegal Activities
- Self-harm
- Harassment
- Unethical Actions
- Social Engineering
- Fraud
- Disinformation
Baseline for every AI system
person: Part of every AI system
Review first
- Overreliance Review priority
Unapproved AI services: Depends on your system
To clarify
Supply chain and technical foundations: 5 baseline threats applying to every AI system
Baseline for every AI system
How it differs from similar use cases
Executes commands itself, edits files and submits complete changes for review
Generative AI assistant for staff
Helps with general drafting, summarising and research rather than programming
Data and information
Processing and operating environment
The language model comes from the assistant's provider or another model provider.
Storage and further use
Many assistants search the organisation's code repository for relevant context. Some companies fine-tune the model on their own code; most use it unchanged.
Results and data sharing
In chat, some assistants call search or development environment functions. With a cloud service, code snippets leave the organisation's environment.
Using a provider does not automatically determine where processing takes place. Storage location, subcontractors, deletion periods and use for training must be checked for the specific deployment.
Assess risks and determine measures
From understanding the system to analysing risks
- Define the system boundary
Record data, participants, interfaces and permissions. Resolve uncertain characteristics in the model.
- Investigate threats
Relate review candidates to actual data flows: what could happen, and who would be affected?
- Assess risk
Assess impacts and likelihood in the deployment context. The number of review candidates is not a risk score.
- Provide evidence of measures
Select appropriate controls, assign responsibilities and define effectiveness checks. Document residual risks and approvals.
Key threats in the application context
- 4 particularly important
- 9 typical
- 27 depending on the system
- 8 additional threats for every AI system
A total of 48 threats from the threat catalogue may be relevant to this use case.
Priority review points
The assistant suggests a nonexistent library, and attackers have published a malicious package under that exact name. Accepting the suggestion introduces malicious code into the organisation's software.
A crafted file in an external project or a dependency comment contains hidden instructions. The assistant then suggests insecure code or discloses its context.
Suggestions that fit smoothly and compile successfully are easily waved through without thorough review. Vulnerabilities such as missing input validation then enter the code.
Sensitive Information Disclosure
The assistant sends snippets from open files to the provider's service. Any credentials or customer data in them leave the organisation's environment.
9 further review candidates in the typical design
Other relevant threats
Training data or an adopted base model can introduce hidden triggers for misbehaviour.
Listed because of this characteristic
- Uses another provider's model
Sleepy Agent (Time/Event-Triggered Hidden Instructions)
An instruction-following model can change behaviour at a particular time or event.
Listed because of this characteristic
- Performs natural language tasks
Users can override the system's working instructions through their inputs.
Listed because of this characteristic
- Performs natural language tasks
People requesting generated content may try to bypass intended limits.
Listed because of this characteristic
- Performs natural language tasks
Carefully crafted queries can reveal system instructions, which must therefore contain no secrets.
Listed because of this characteristic
- Performs natural language tasks
Complex or high-volume inputs can exhaust model computing resources.
Listed because of this characteristic
- Generates its own text
Targeted queries can recover some personal or protected training data from the model.
Listed because of this characteristic
- Processes confidential data
Exfiltration from ML Application
Protected data can leak through tools, network connections, storage or insiders.
Listed because of this characteristic
- Processes confidential data
Factual Inconsistencies (Hallucinations)
Generated text can invent facts or distort them during rewriting.
Listed because of this characteristic
- Generates its own text
Up to 27 further threats may apply, depending on your system design. Nine threats need to be reviewed for every AI system. One of these is particularly important here and is therefore listed above.
System features and open questions
Each Yes adds further threats to your checklist, up to 27 in total. If you cannot yet answer a question, include its threats in your review.
- Added by this Yes
- Also requires a Yes to another question
How your system is built
- Generates content from open-ended requests10 threats, plus 2 only in combination with other characteristics
- Uses the Model Context Protocol (MCP)3 threats
- Calls other programs or interfaces2 threats
- Uses operational feedback2 threats
- Learns from your own data1 threat, plus 1 only in combination with other characteristics
- Searches its own document collection1 threat
- Formulates answers from retrieved documents1 threat
- Remembers content beyond a single conversation1 threat
- Learns from examples with correct answers1 threat
- Processes text, images and speech together1 threat
How you operate the system
- Available to people outside the organisation4 threats, plus 2 only in combination with other characteristics
- Accepts queries directly through an interface4 threats
- Accessible over a network1 threat, plus 1 only in combination with other characteristics
- Incurs costs with each query1 threat
- Shares computing resources with other customers or applications1 threat
- Used alongside unapproved AI services1 threat
How your system is built
Does the system create new content from open-ended requests that people read, see or hear directly? Translation, reading aloud and transcription do not count.
A chat-enabled assistant also answers open-ended questions; basic code completion does not.
If yes, add
Does the system search its own collection of documents or records for relevant content?
Many assistants search the organisation's code repository for relevant context.
If yes, add
Does the system formulate answers from retrieved documents?
If yes, add
Does the system remember content from earlier conversations or tasks and reuse it later?
If yes, add
Does the system call other programs, interfaces or search services itself, even if only to read?
In chat, some assistants call search or development environment functions.
If yes, add
Is the system connected to tools or data sources through the Model Context Protocol (MCP)?
Development environments increasingly connect assistants to other tools through the Model Context Protocol (MCP).
If yes, add
Does your organisation train or fine-tune the model itself using its own data?
Some companies fine-tune the model on their own code; most use it unchanged.
If yes, add
Is the model trained on examples to which someone has assigned the correct answer?
Whether the provider fine-tuned the model on labelled examples usually becomes clear only from its documentation.
If yes, add
Does operational feedback or output later feed into analysis or training?
Accepted and rejected suggestions may feed into the provider's further development.
If yes, add
Does the system process several input types together, such as text with images or speech?
If yes, add
How you operate the system
These questions arise for almost every AI system.
Can people or programs query the model directly, for example through an interface?
If yes, add
Is the system accessible over a network or dependent on network services?
If yes, add
Do you pay per query or unit of computing time, so costs increase with use?
If yes, add
Does the system share servers or computing resources with other customers or applications?
If yes, add
Can people outside your organisation use the system, including without login?
If yes, add
Do staff also use AI services for this task that your organisation has not approved?
Some staff use private accounts or unapproved extensions.
If yes, add
Controls and implementation
Measures for the priority review areas are expanded. Each linked measure explains how to implement it.
Review priorityMalicious Software3 measures
Review priorityPrompt Injection: Indirect3 measures
Review priorityOverreliance3 measures
Review prioritySensitive Information Disclosure3 measures
Backdoor ML Model2 measures
Sleepy Agent (Time/Event-Triggered Hidden Instructions)3 measures
Prompt Injection: Direct3 measures
Jailbreaks2 measures
Meta Prompt Extraction2 measures
Model Denial of Service3 measures
Privacy Attacks3 measures
Exfiltration from ML Application3 measures
Factual Inconsistencies (Hallucinations)3 measures
Controls for the remaining threats are available on their pages in the threat catalogue.
AI Act provisions
-
Art. 4: AI literacy
For every AI system Applies since 2 February 2025.
-
Art. 53 para. 1 point b, Annex XII: Embedded general-purpose model
Depending on use Review trigger since 2 August 2025.
Applies to every AI system
Art. 4: AI literacy
Applies if staff or appointed representatives operate or use AI systems.
Applies since 2 February 2025.
Obligations and details for Art. 4: AI literacy
- Who bears the obligations
- Provider, Deployer, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
May apply in particular uses
Art. 53 para. 1 point b, Annex XII: Embedded general-purpose model
Applies if a large language model or another third-party general-purpose model is incorporated into an assistant you provide yourself.
Applies since 2 August 2025.
Obligations and details for Art. 53 para. 1 point b, Annex XII: Embedded general-purpose model
- The general condition is
- Applies if another provider's general-purpose AI model is incorporated, such as a large language model.
- Points to consider
- The model provider bears its obligations. It must give you the Annex XII information; use it for your own documentation.
- Who bears the obligations
- Provider of a general-purpose AI model
Detailed obligations
Classify your system with the quick checker All obligations in the obligations compass
Cross-cutting threats
Review these threats for every AI system, whatever its purpose.
Atypical system features
View 4 further questions
If any of these features does apply to your system, add the listed threats.
Does the system use a knowledge graph storing concepts and their relationships?
If yes, add
Does the system determine its next steps itself based on earlier results?
The assistant handles individual instructions and does not plan its own sequence of tasks.
If yes, add
Do documents from several customers, departments or permission groups share one search store?
If yes, add
Is the system intended to maximise a specified metric, such as revenue, utilisation or clicks?
If yes, add
Sources
- Authority, court or parliament
BSI and ANSSI, AI Coding Assistants, 4 October 2024
The joint German-French publication describes the opportunities and security risks of coding assistants and recommends safeguards.
- Manufacturer
GitHub, Mercedes-Benz customer story According to the manufacturer
The provider reports that more than 5,000 Mercedes-Benz developers use GitHub Copilot.
- Report of a deployment
Federal Employment Agency, The agency's current main AI applications, 26 January 2026 According to the deployer
The agency lists Tabnine as a coding assistant for all its IT developers.
- Register or case collection
Dutch Algorithm Register, GitHub Copilot, Zeeuws Archief
The register lists GitHub Copilot as an operational tool that suggests code during programming.
- Specialist study
The study shows that language models invent package names that attackers can register and populate with malicious code.
Related use cases
Also in the field of Software development and IT security
- Coding agentThe agent takes on complete programming tasks, works in the code repository and terminal, executes commands itself and submits changes for review.
- AI-based cyberattack detectionThe system learns normal patterns in networks, devices, accounts and email traffic, then flags deviations that may indicate attacks, phishing or compromised accounts.
Updated: 30 September 2026. Review checklist rules version 0.3.2, threat catalogue version 2026.10.24.2. How these pages are structured