Input data and information sources
Incoming emails, invitations and shared documents also come from outsiders; EchoLeak required only a crafted email. Mailboxes, calendars and chats contain names, contact details and conversation content.
AI assistant in office applications
Also known as: Embedded workplace AI assistant (copilot)
Staff invoke the assistant directly in email, word processing, spreadsheets and chat; it accesses mailboxes and file storage using their permissions.
Understand the use case
What defines this use case?
The assistant drafts emails, summarises meetings and documents, and answers questions about anything the user is allowed to see. The UK government has trialled it, an IT services company is introducing it, and a Dutch municipality already uses it.
This page describes a typical system design. Actual data flows and permissions depend on your implementation.
- System design
- Knowledge assistant grounded in sources
- Ability to act
- Action permissions depend on integration
- Fields of use
- public administration, IT services
- Also known as
- Copilot, Microsoft 365 Copilot, AI in email and word processing
How it works, step by step
System design and data flows
Input data
A user asks a question or gives an instruction in email, word processing, a spreadsheet or chat.
Incoming emails, invitations and shared documents also come from outsiders; EchoLeak required only a crafted email. Mailboxes, calendars and chats contain names, contact details and conversation content.
Processing
The assistant searches mailboxes, calendars and file storage using the user's permissions and passes the results to a language model.
The office application supplies the model with the instruction and available working context, such as an open document or a selected message. The model produces a suggestion within that application. Which other files are included depends on the connected services and permissions.
Results
The assistant provides a draft, summary or answer with references to the content used.
A generated result is newly worded or created. To assess it, the instruction and context on which it is based must remain identifiable.
Use and impact
The user sends or saves the output, incorporating it into emails, documents and decisions.
Integration determines whether the result is simply displayed, adopted after approval or written directly into a business application. These variants differ in their data flows and permissions.
Explore the system design: Select a component to see its task, incoming and outgoing data, and role in the use case. The guided walkthrough explains each phase. Then use Yes, Uncertain and No to represent your specific system variant.
23 review first17 clarify9 baseline threats
- Present
- Depends on your system
- Absent
- Review first
- To clarify
How to use the model
The diagram shows the typical architecture. Select a component to see what happens there and adapt its features to your system. Enable “Analysis guidance” to see which threats need reviewing for this architecture. The page's checklist continues to describe the typical case.
Changes from the typical case
None yet. Each change will appear here.
Where most threats arise
- 12 review first, 2 clarify
- 4 review first
- 3 review first, 1 clarify
- 4 clarify
In this system design: 23 to review first, 17 to clarify, 9 baseline threats.
Architecture as a list
External access: Depends on your system
To clarify
Input: Part of every AI system
Review first
To clarify
Baseline for every AI system
Third-party content: Present
Review first
- Prompt Injection: Indirect Review priority
AI model: Part of every AI system
Review first
- Sleepy Agent (Time/Event-Triggered Hidden Instructions)
- Prompt Injection: Direct
- Jailbreaks
- Meta Prompt Extraction
Baseline for every AI system
Provider service: Present
Review first
To clarify
Document collection: Present
Review first
- RAG-Specific Attacks (Document Poisoning) Review priority
- Cross-Tenant Leakage (Multi-Tenant Vector DB) Review priority
To clarify
Memory: Depends on your system
To clarify
Training and feedback: Depends on your system
To clarify
Tools and interfaces: Depends on your system
To clarify
Result: Part of every AI system
Review first
- Model Denial of Service
- Hate Speech and Discrimination
- Profanity
- Violence / Unsafe Actions
- Controversial Topics
- Illegal Activities
- Harassment
- Unethical Actions
- Social Engineering
- Fraud
- Disinformation
- Factual Inconsistencies (Hallucinations)
To clarify
Baseline for every AI system
Business application or equipment: Depends on your system
person: Part of every AI system
Baseline for every AI system
Affected person: Depends on your system
Unapproved AI services: Depends on your system
To clarify
Supply chain and technical foundations: 5 baseline threats applying to every AI system
Baseline for every AI system
How it differs from similar use cases
Generative AI assistant for staff
Works only with staff inputs, without access to mailboxes or storage
AI knowledge assistant (enterprise search)
Answers from a defined, editorially maintained document collection
Completes tasks independently in several steps, with its own permissions in business applications
Data and information
Processing and operating environment
The office application provider supplies the language model and search service.
Storage and further use
The assistant searches mailboxes, calendars, chats and file storage for relevant content. It formulates answers from the emails and documents it finds.
Results and data sharing
Depending on the product, the assistant calls functions in other applications or extensions. Depending on the product and settings, the assistant sends emails, creates appointments and edits files itself, or only provides drafts.
Using a provider does not automatically determine where processing takes place. Storage location, subcontractors, deletion periods and use for training must be checked for the specific deployment.
Assess risks and determine measures
From understanding the system to analysing risks
- Define the system boundary
Record data, participants, interfaces and permissions. Resolve uncertain characteristics in the model.
- Investigate threats
Relate review candidates to actual data flows: what could happen, and who would be affected?
- Assess risk
Assess impacts and likelihood in the deployment context. The number of review candidates is not a risk score.
- Provide evidence of measures
Select appropriate controls, assign responsibilities and define effectiveness checks. Document residual risks and approvals.
Key threats in the application context
- 4 particularly important
- 19 typical
- 17 depending on the system
- 9 for every AI system
A total of 49 threats from the threat catalogue may be relevant to this use case.
Priority review points
An outsider sends an email containing hidden instructions that the assistant reads during its next search. It follows them even though the user has never opened the email.
Exfiltration from ML Application
Injected instructions make the assistant embed internal content in a link or image. When the application loads the image, the data reaches an external server, as happened with EchoLeak.
Cross-Tenant Leakage (Multi-Tenant Vector DB)
Folders containing personnel or board records are accidentally shared with everyone. The assistant finds them and reveals their contents to anyone who asks.
RAG-Specific Attacks (Document Poisoning)
Someone places a document with false information in shared storage. The assistant cites it as a source and spreads the error in answers to many staff members.
19 further review candidates in the typical design
Other relevant threats
Training data or an adopted base model can introduce hidden triggers for misbehaviour.
Listed because of this characteristic
- Uses another provider's model
Sleepy Agent (Time/Event-Triggered Hidden Instructions)
An instruction-following model can change behaviour at a particular time or event.
Listed because of this characteristic
- Performs natural language tasks
Users can override the system's working instructions through their inputs.
Listed because of this characteristic
- Performs natural language tasks
People requesting generated content may try to bypass intended limits.
Listed because of this characteristic
- Performs natural language tasks
Carefully crafted queries can reveal system instructions, which must therefore contain no secrets.
Listed because of this characteristic
- Performs natural language tasks
Complex or high-volume inputs can exhaust model computing resources.
Listed because of this characteristic
- Generates its own text
Targeted queries can recover some personal or protected training data from the model.
Listed because of these characteristics
- Processes personal data
- Processes confidential data
Sensitive Information Disclosure
Confidential content can appear in answers, supplied context or logs.
Listed because of these characteristics
- Processes personal data
- Processes confidential data
Hate Speech and Discrimination
Open-ended generated content can contain discriminatory or hateful statements.
Listed because of these characteristics
- Generates its own text
- Generates content from open-ended requests
Freely generated answers can use an inappropriate tone.
Listed because of these characteristics
- Generates its own text
- Generates content from open-ended requests
Open-ended content can depict violence or assist dangerous actions.
Listed because of these characteristics
- Generates its own text
- Generates content from open-ended requests
Answers on controversial topics may require separate editorial review.
Listed because of these characteristics
- Generates its own text
- Generates content from open-ended requests
Open-ended content can facilitate unlawful acts. The required scrutiny depends on purpose and users.
Listed because of these characteristics
- Generates its own text
- Generates content from open-ended requests
Open-ended content can be used for harassment.
Listed because of these characteristics
- Generates its own text
- Generates content from open-ended requests
Generated suggestions can cross ethical boundaries in the specific context.
Listed because of these characteristics
- Generates its own text
- Generates content from open-ended requests
Generated images, voices and texts can deceive or impersonate people.
Listed because of these characteristics
- Generates its own text
- Generates content from open-ended requests
Generated content can facilitate fraud, for example through forged messages or documents.
Listed because of these characteristics
- Generates its own text
- Generates content from open-ended requests
Synthetic content can be presented as authentic and spread misleadingly.
Listed because of these characteristics
- Generates its own text
- Generates content from open-ended requests
Factual Inconsistencies (Hallucinations)
Generated text can invent facts or distort them during rewriting.
Listed because of this characteristic
- Generates its own text
Up to 17 further threats may apply, depending on your system design. Nine threats need to be reviewed for every AI system.
System features and open questions
Each Yes adds further threats to your checklist, up to 17 in total. If you cannot yet answer a question, include its threats in your review.
- Added by this Yes
How your system is built
- Uses the Model Context Protocol (MCP)3 threats
- Calls other programs or interfaces2 threats
- Uses operational feedback2 threats
- Generates images, audio or videos1 threat
- Uses a knowledge graph1 threat
- Remembers content beyond a single conversation1 threat
- Changes data or triggers processes1 threat
- Processes text, images and speech together1 threat
- Generates code or queries1 threat
- Executes programs or commands1 threat
How you operate the system
How your system is built
Does the system generate or edit images, sounds, voices or videos?
In some office applications, the assistant also generates images, for example for presentations.
If yes, add
Does the system use a knowledge graph storing concepts and their relationships?
If yes, add
Does the system remember content from earlier conversations or tasks and reuse it later?
Some products remember preferences and information from earlier conversations.
If yes, add
Does the system call other programs, interfaces or search services itself, even if only to read?
Depending on the product, the assistant calls functions in other applications or extensions.
If yes, add
Can the system change things itself, such as storing data, sending messages, initiating transactions or granting permissions?
Depending on the product and settings, the assistant sends emails, creates appointments and edits files itself, or only provides drafts.
If yes, add
Is the system connected to tools or data sources through the Model Context Protocol (MCP)?
If yes, add
Does operational feedback or output later feed into analysis or training?
If yes, add
Does the system process several input types together, such as text with images or speech?
If yes, add
Does the system generate code, scripts or database queries for further use?
In spreadsheets, the assistant also writes formulas and scripts.
If yes, add
Does the system execute programs, scripts or commands itself?
If yes, add
How you operate the system
These questions arise for almost every AI system.
Can people or programs query the model directly, for example through an interface?
If yes, add
Is the system accessible over a network or dependent on network services?
If yes, add
Do you pay per query or unit of computing time, so costs increase with use?
If yes, add
Does the system share servers or computing resources with other customers or applications?
If yes, add
Do staff also use AI services for this task that your organisation has not approved?
If yes, add
Controls and implementation
Measures for the priority review areas are expanded. Each linked measure explains how to implement it.
Review priorityPrompt Injection: Indirect3 measures
Review priorityExfiltration from ML Application3 measures
Review priorityCross-Tenant Leakage (Multi-Tenant Vector DB)3 measures
Review priorityRAG-Specific Attacks (Document Poisoning)3 measures
Backdoor ML Model2 measures
Sleepy Agent (Time/Event-Triggered Hidden Instructions)3 measures
Prompt Injection: Direct3 measures
Jailbreaks2 measures
Meta Prompt Extraction2 measures
Model Denial of Service3 measures
Privacy Attacks3 measures
Sensitive Information Disclosure3 measures
Hate Speech and Discrimination2 measures
Profanity2 measures
Violence / Unsafe Actions2 measures
Controversial Topics3 measures
Illegal Activities3 measures
Harassment3 measures
Unethical Actions3 measures
Social Engineering3 measures
Fraud3 measures
Disinformation2 measures
Factual Inconsistencies (Hallucinations)3 measures
Controls for the remaining threats are available on their pages in the threat catalogue.
AI Act provisions
-
Art. 4: AI literacy
For every AI system Applies since 2 February 2025.
-
Art. 50 para. 4: Labelling deepfakes and published texts
Depending on use Review trigger since 2 August 2026.
Applies to every AI system
Art. 4: AI literacy
Applies if staff or appointed representatives operate or use AI systems.
Applies since 2 February 2025.
Obligations and details for Art. 4: AI literacy
- Who bears the obligations
- Provider, Deployer, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
May apply in particular uses
Art. 50 para. 4: Labelling deepfakes and published texts
Applies if texts produced with the assistant are published on matters of public interest without human editorial review and responsibility.
Applies since 2 August 2026.
Obligations and details for Art. 50 para. 4: Labelling deepfakes and published texts
- The general condition is
- Applies if the system creates deepfakes or AI texts on matters of public interest are published without editorial responsibility.
- Who bears the obligations
- Deployer
Detailed obligations
Generally does not apply
Art. 50 para. 1: Disclosure of AI interaction
Staff knowingly invoke the assistant in their office application, making it obvious that they are using an AI system.
Obligations and details for Art. 50 para. 1: Disclosure of AI interaction
- The general condition is
- Applies if people directly write or speak with the system and it is not obvious that they are interacting with AI.
- Who bears the obligations
- Provider, Product manufacturer
Detailed obligations
Classify your system with the quick checker All obligations in the obligations compass
Cross-cutting threats
Review these threats for every AI system, whatever its purpose.
Atypical system features
View 3 further questions
If any of these features does apply to your system, add the listed threats.
Does the system determine its next steps itself based on earlier results?
The assistant handles individual instructions and does not plan its own sequence of tasks.
If yes, add
Can people outside your organisation use the system, including without login?
If yes, add
Is the system intended to maximise a specified metric, such as revenue, utilisation or clicks?
If yes, add
Sources
- Authority, court or parliament
GOV.UK, Microsoft 365 Copilot Experiment: Cross-Government Findings Report
Twelve UK government organisations trialled the assistant with 20,000 staff from September to December 2024.
- Report of a deployment
Bechtle AG, Bechtle introduces Microsoft 365 Copilot for 5,000 employees, Press release, 2024 According to the deployer
The company states that it is introducing the assistant for 5,000 staff.
- Register or case collection
Dutch Algorithm Register, Microsoft M365 Copilot, Gemeente Raalte
The municipality has used the assistant since January 2025. It accesses documents, emails, calendars and chats visible to the signed-in user.
- Specialist study
Checkmarx, EchoLeak (CVE-2025-32711) Shows Us That AI Security Is Challenging
A crafted email in a mailbox was enough to make Microsoft 365 Copilot disclose internal data without user involvement. The vulnerability is recorded as CVE-2025-32711.
- Register or case collection
MITRE ATLAS, atlas-data CHANGELOG
Case studies AML.CS0026 and AML.CS0059 cover assistants of this kind.
Related use cases
Also in the field of Office work and knowledge management
- Generative AI assistant for staffStaff draft, revise, translate and shorten texts using a language model their organisation provides in a secured environment.
- AI knowledge assistant (enterprise search)The assistant answers questions from a defined document collection and shows which documents support the answer.
- Speech recognition and transcriptionSpeech recognition automatically transcribes dictation, conversations and other recordings.
- AI meeting minutes and conversation documentationFrom recorded meetings and consultation conversations, the system drafts minutes or a case note with an action list.
- Machine translationPublic authorities and businesses automatically translate texts, websites and documents, often using their own glossary.
- AI agentAn AI agent breaks a task into steps and completes it largely independently using tools, interfaces and its own permissions in business applications.
Updated: 30 September 2026. Review checklist rules version 0.3.2, threat catalogue version 2026.10.24.2. How these pages are structured