Input data and information sources
Every input comes from someone outside the organisation. Callers and chat users provide names, customer numbers, addresses and the details of their request.
Customer service chatbots and voicebots
Also known as: Customer service chatbot, voicebot
A bot independently handles customers' or citizens' requests in chat or by phone and passes difficult cases to people.
Understand the use case
What defines this use case?
Telecommunications providers, payment services and public authorities use these bots. BaFin lists chatbots among banks' most common AI applications.
This page describes a typical system design. Actual data flows and permissions depend on your implementation.
- System design
- Analysis and processing
- Ability to act
- Action permissions depend on integration
- Fields of use
- telecommunications, banking, payment services, public administration, aviation
- Also known as
- virtual assistant, voice dialogue system, conversational IVR, AI customer assistant
How it works, step by step
System design and data flows
Input data
A customer describes their request in their own words in a chat window or by phone.
Every input comes from someone outside the organisation. Callers and chat users provide names, customer numbers, addresses and the details of their request.
Processing
The bot identifies the request, finds the relevant information and retrieves customer data if needed.
The system processes a message or first converts speech into text. Dialogue management and, where applicable, knowledge retrieval provide context; the model generates the answer. A voicebot then converts it back into speech. Access to customer data or booking systems involves additional functions.
Results
The bot responds in writing or with a synthetic voice, handles simple transactions or connects the user to a person.
The result describes the defined model task. A match, category or forecast acquires its professional significance only in conjunction with the underlying data and intended use.
Use and impact
The customer acts on the information, and the bot's commitments may bind the deployer.
Integration determines whether the result is simply displayed, adopted after approval or written directly into a business application. These variants differ in their data flows and permissions.
Explore the system design: Select a component to see its task, incoming and outgoing data, and role in the use case. The guided walkthrough explains each phase. Then use Yes, Uncertain and No to represent your specific system variant.
15 review first23 clarify7 baseline threats
- Present
- Depends on your system
- Absent
- Review first
- To clarify
How to use the model
The diagram shows the typical architecture. Select a component to see what happens there and adapt its features to your system. Enable “Analysis guidance” to see which threats need reviewing for this architecture. The page's checklist continues to describe the typical case.
Changes from the typical case
None yet. Each change will appear here.
Where most threats arise
- 1 review first, 13 clarify
- 5 review first, 1 clarify
- 4 review first
- 2 review first, 1 clarify
In this system design: 15 to review first, 23 to clarify, 7 baseline threats.
Architecture as a list
External access: Present
Review first
- Cost Harvesting / Repurposing Review priority
- Model Theft
- Model Denial of Service
- Application Denial of Service
- Model Reconnaissance
To clarify
Input: Part of every AI system
Review first
To clarify
Baseline for every AI system
Third-party content: Present
Review first
AI model: Part of every AI system
Review first
- Prompt Injection: Direct Review priority
- Sleepy Agent (Time/Event-Triggered Hidden Instructions)
- Jailbreaks
- Meta Prompt Extraction
Baseline for every AI system
Provider service: Present
Review first
To clarify
Document collection: Depends on your system
To clarify
Memory: Depends on your system
To clarify
Training and feedback: Depends on your system
To clarify
Tools and interfaces: Depends on your system
To clarify
Result: Part of every AI system
Review first
- Insecure Output Handling Review priority
To clarify
Business application or equipment: Depends on your system
person: Part of every AI system
Review first
- Overreliance Review priority
Affected person: Depends on your system
Unapproved AI services: Depends on your system
To clarify
Supply chain and technical foundations: 5 baseline threats applying to every AI system
Baseline for every AI system
How it differs from similar use cases
Completes requests independently in several steps, with its own permissions in business applications
AI knowledge assistant (enterprise search)
Answers staff questions from an internal document collection
Intelligent document processing (IDP)
Only routes written requests to the responsible team, without answering them
Data and information
Processing and operating environment
Language models and speech recognition generally come from providers; Amsterdam uses PolyAI.
Storage and further use
Many bots search a question-and-answer knowledge base or help pages. Some bots access earlier conversations with the same person.
Results and data sharing
Some bots retrieve customer data, check outages or book appointments through interfaces. A bot that changes tariffs, addresses or appointments alters customer records; other bots only provide information.
Using a provider does not automatically determine where processing takes place. Storage location, subcontractors, deletion periods and use for training must be checked for the specific deployment.
Assess risks and determine measures
From understanding the system to analysing risks
- Define the system boundary
Record data, participants, interfaces and permissions. Resolve uncertain characteristics in the model.
- Investigate threats
Relate review candidates to actual data flows: what could happen, and who would be affected?
- Assess risk
Assess impacts and likelihood in the deployment context. The number of review candidates is not a risk score.
- Provide evidence of measures
Select appropriate controls, assign responsibilities and define effectiveness checks. Document residual risks and approvals.
Key threats in the application context
- 4 particularly important
- 11 typical
- 23 depending on the system
- 7 additional threats for every AI system
A total of 45 threats from the threat catalogue may be relevant to this use case.
Priority review points
Someone tells the bot to ignore its instructions and confirm a discount. If it complies, the conversation contains a commitment the person can rely on.
The bot's answer reaches the customer directly without review. If it gives an incorrect deadline or refund rule, the customer acts on it and the deployer must answer for it, as in the Air Canada case.
The bot requires no login and calls a paid language model for every message. Large volumes of automated requests drive up costs or exploit the bot as free model access.
Customers treat the bot's information as binding because it speaks for the company. Staff accept its summary of a request without reading the conversation.
11 further review candidates in the typical design
Other relevant threats
Training data or an adopted base model can introduce hidden triggers for misbehaviour.
Listed because of this characteristic
- Uses another provider's model
Sleepy Agent (Time/Event-Triggered Hidden Instructions)
An instruction-following model can change behaviour at a particular time or event.
Listed because of this characteristic
- Performs natural language tasks
Anyone able to query the model can replicate it through many queries. Internally trained weights can be stolen through network access.
Listed because of this characteristic
- Available to people outside the organisation
A model can follow instructions in a document or tool result as if they were its own working instructions.
Listed because of these characteristics
- Performs natural language tasks
- Processes content outsiders can write
People requesting generated content may try to bypass intended limits.
Listed because of this characteristic
- Performs natural language tasks
Carefully crafted queries can reveal system instructions, which must therefore contain no secrets.
Listed because of this characteristic
- Performs natural language tasks
Complex or high-volume inputs can exhaust model computing resources.
Listed because of this characteristic
- Available to people outside the organisation
Targeted queries can recover some personal or protected training data from the model.
Listed because of this characteristic
- Processes personal data
Sensitive Information Disclosure
Confidential content can appear in answers, supplied context or logs.
Listed because of this characteristic
- Processes personal data
A network-accessible application can be overwhelmed by mass requests.
Listed because of this characteristic
- Available to people outside the organisation
Targeted queries can reveal the model's architecture and safeguards.
Listed because of this characteristic
- Available to people outside the organisation
Up to 23 further threats may apply, depending on your system design. Nine threats need to be reviewed for every AI system. Two of these are particularly important here and are therefore listed above.
System features and open questions
Each Yes adds further threats to your checklist, up to 23 in total. If you cannot yet answer a question, include its threats in your review.
- Added by this Yes
- Also requires a Yes to another question
How your system is built
- Generates images, audio or videos3 threats, plus 7 only in combination with other characteristics
- Calls other programs or interfaces3 threats
- Uses operational feedback2 threats
- Generates its own text1 threat, plus 13 only in combination with other characteristics
- Searches its own document collection1 threat
- Formulates answers from retrieved documents1 threat
- Remembers content beyond a single conversation1 threat
- Changes data or triggers processes1 threat
- Processes confidential data1 threat
- Learns from your own data1 threat
- Learns from examples with correct answers1 threat
- Processes text, images and speech together1 threat
- Generates content from open-ended requests12 threats only in combination with other characteristics
How you operate the system
- Passes content to services outside your environment1 threat
- Shares computing resources with other customers or applications1 threat
- Used alongside unapproved AI services1 threat
- Accepts queries directly through an interface1 threat only in combination with other characteristics
How your system is built
Does the system generate its own text, such as answers, drafts or summaries?
Older bots select prepared responses; language-model bots generate their own, as Telekom describes for “Frag Magenta”.
If yes, add
Does the system generate or edit images, sounds, voices or videos?
Voicebots often speak with a synthetic voice.
If yes, add
Does the system create new content from open-ended requests that people read, see or hear directly? Translation, reading aloud and transcription do not count.
A bot restricted to approved information does not create open-ended content; an unrestricted language model does.
If yes, add
Does the system search its own collection of documents or records for relevant content?
Many bots search a question-and-answer knowledge base or help pages.
If yes, add
Does the system formulate answers from retrieved documents?
If yes, add
Does the system remember content from earlier conversations or tasks and reuse it later?
Some bots access earlier conversations with the same person.
If yes, add
Does the system call other programs, interfaces or search services itself, even if only to read?
Some bots retrieve customer data, check outages or book appointments through interfaces.
If yes, add
Can the system change things itself, such as storing data, sending messages, initiating transactions or granting permissions?
A bot that changes tariffs, addresses or appointments alters customer records; other bots only provide information.
If yes, add
Does the system process confidential information such as trade secrets, health or financial data?
Banks and insurers handle accounts and contracts; public authorities handle benefits and procedures.
If yes, add
Does your organisation train or fine-tune the model itself using its own data?
If yes, add
Is the model trained on examples to which someone has assigned the correct answer?
Intent recognition often learns from example sentences labelled with the correct intent.
If yes, add
Does operational feedback or output later feed into analysis or training?
Abandoned or transferred conversations often provide material for improving the bot.
If yes, add
Does the system process several input types together, such as text with images or speech?
If yes, add
How you operate the system
These questions arise for almost every AI system.
Can people or programs query the model directly, for example through an interface?
If yes, add
Can content leave your environment, for example to a cloud provider, web search or email?
If yes, add
Does the system share servers or computing resources with other customers or applications?
If yes, add
Do staff also use AI services for this task that your organisation has not approved?
If yes, add
Controls and implementation
Measures for the priority review areas are expanded. Each linked measure explains how to implement it.
Review priorityPrompt Injection: Direct3 measures
Review priorityInsecure Output Handling3 measures
Review priorityCost Harvesting / Repurposing3 measures
Review priorityOverreliance3 measures
Backdoor ML Model2 measures
Sleepy Agent (Time/Event-Triggered Hidden Instructions)3 measures
Model Theft3 measures
Prompt Injection: Indirect3 measures
Jailbreaks2 measures
Meta Prompt Extraction2 measures
Model Denial of Service3 measures
Privacy Attacks3 measures
Sensitive Information Disclosure3 measures
Application Denial of Service3 measures
Model Reconnaissance3 measures
Controls for the remaining threats are available on their pages in the threat catalogue.
AI Act provisions
-
Art. 4: AI literacy
For every AI system Applies since 2 February 2025.
-
Art. 53 para. 1 point b, Annex XII: Embedded general-purpose model
Depending on use Review trigger since 2 August 2025.
-
Art. 50 para. 1: Disclosure of AI interaction
In typical use Review trigger since 2 August 2026.
-
Annex III no. 5 point a: Public assistance benefits
Depending on use Review trigger from 2 December 2027.
Applies to every AI system
Art. 4: AI literacy
Applies if staff or appointed representatives operate or use AI systems.
Applies since 2 February 2025.
Obligations and details for Art. 4: AI literacy
- Who bears the obligations
- Provider, Deployer, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
May apply in typical use
Art. 50 para. 1: Disclosure of AI interaction
Applies if customers or citizens chat or speak with the bot and it is not obvious that an AI is responding.
Applies since 2 August 2026.
Obligations and details for Art. 50 para. 1: Disclosure of AI interaction
- The general condition is
- Applies if people directly write or speak with the system and it is not obvious that they are interacting with AI.
- Who bears the obligations
- Provider, Product manufacturer
Detailed obligations
May apply in particular uses
Annex III no. 5 point a: Public assistance benefits
Applies if the bot assesses eligibility for a public benefit on behalf of an authority, rather than merely providing information or routing requests.
Applies from 2 December 2027.
Obligations and details for Annex III no. 5 point a: Public assistance benefits
- The general condition is
- Applies if authorities or bodies acting for them use the system to assess eligibility for public benefits and services, or to grant, reduce or recover them.
- Who bears the obligations
- Provider, Deployer, Product manufacturer
Detailed obligations
Art. 53 para. 1 point b, Annex XII: Embedded general-purpose model
Applies if your organisation builds the bot itself using a large language model or another third-party general-purpose model.
Applies since 2 August 2025.
Obligations and details for Art. 53 para. 1 point b, Annex XII: Embedded general-purpose model
- The general condition is
- Applies if another provider's general-purpose AI model is incorporated, such as a large language model.
- Points to consider
- The model provider bears its obligations. It must give you the Annex XII information; use it for your own documentation.
- Who bears the obligations
- Provider of a general-purpose AI model
Detailed obligations
Classify your system with the quick checker All obligations in the obligations compass
Cross-cutting threats
Review these threats for every AI system, whatever its purpose.
Atypical system features
View 7 further questions
If any of these features does apply to your system, add the listed threats.
Does the system use a knowledge graph storing concepts and their relationships?
If yes, add
Does the system determine its next steps itself based on earlier results?
The bot responds in conversation and does not plan its own steps in business applications.
If yes, add
Does the system delegate tasks to other AI agents or incorporate their results?
If yes, add
Is the system connected to tools or data sources through the Model Context Protocol (MCP)?
If yes, add
Do documents from several customers, departments or permission groups share one search store?
If yes, add
Does the system generate code, scripts or database queries for further use?
If yes, add
Is the system intended to maximise a specified metric, such as revenue, utilisation or clicks?
If yes, add
Sources
- Report of a deployment
Deutsche Telekom, Our chatbot helps around the clock According to the deployer
Telekom states that its chatbot and voice portal are operational and use generative AI.
- Report of a deployment
Klarna, Press release, 27 February 2024 According to the deployer
Klarna states that its AI assistant handled 2.3 million customer conversations in its first month.
- Register or case collection
UK Algorithmic Transparency Record, DVLA Contact Centre Chatbot service
The UK's Driver and Vehicle Licensing Agency operates a chatbot on its website.
- Register or case collection
Dutch Algorithm Register, PolyAI Voicebot, Gemeente Amsterdam
A voicebot identifies callers' needs and routes them onwards.
- Authority, court or parliament
BaFin, Risks in Focus 2026: Digitalisation
BaFin lists chatbot customer service among banks' most common AI applications.
- Register or case collection
AI Incident Database, Incident 639 (Air Canada)
Air Canada was held responsible for incorrect information provided by its chatbot.
- Specialist study
European Commission Joint Research Centre (JRC), AI Watch: Beyond pilots (JRC126665)
Chatbots and virtual agents are among the most common AI applications in the public sector.
Related use cases
Also in the field of Customer service, marketing and platforms
- Recommender systemThe system uses a person's behaviour and profile to select products, content or search results and determine their order.
- Dynamic pricingOnline retailers and platforms automatically adjust prices to demand, costs, capacity and competitors' prices.
- Personalised online advertising (behavioural targeting)The system uses a person's behaviour and profile to select which advertisements they see.
- Automated content moderationPlatforms automatically identify unlawful content or breaches of their rules and remove, restrict or refer posts for human review.
- Image, video and voice generationThe system generates or edits images, videos and audio from a short description, for example for advertising and communication.
Often used together with
Updated: 30 September 2026. Review checklist rules version 0.3.2, threat catalogue version 2026.10.24.2. How these pages are structured