Versatile AI Risk Assessment
Office work and knowledge management

Generative AI assistant for staff

Also known as: Enterprise generative AI assistant

Staff draft, revise, translate and shorten texts using a language model their organisation provides in a secured environment.

Explore the interactive model

Export handout

A concise PDF covering how the system works, its data flows and points to investigate, ready to share.

The explanations and measures describe the typical system design. Extensive personal changes add an appendix.

Understand the use case

What defines this use case?

They give instructions in natural language and receive drafts to review and reuse themselves. A German state administration, the Federal Employment Agency and a city administration use these assistants.

This page describes a typical system design. Actual data flows and permissions depend on your implementation.

System design
Generative assistance
Ability to act
Provides results for subsequent use
Fields of use
public administration
Also known as
AI chat for staff, enterprise ChatGPT, writing assistant

How it works, step by step

System design and data flows

  1. Input data

    Staff enter an instruction and attach texts or files if needed.

    Staff paste third-party emails, websites and documents for processing. Many assistants accept images, tables or speech as well as text.

  2. Processing

    A language model reads the instruction and material and formulates an answer.

    The language model processes the instruction together with the supplied context and generates the text step by step. It does not automatically distinguish a substantiated statement from plausible wording. Searching approved sources is an additional processing step, where this capability is integrated.

  3. Results

    It returns a draft, summary or translation.

    A generated result is newly worded or created. To assess it, the instruction and context on which it is based must remain identifiable.

  4. Use and impact

    Staff incorporate the text into letters, notes or presentations used internally or externally.

    The model supplies information for subsequent use. Selection, adoption or a professional decision in the following workflow is a separate step and should not automatically be attributed to the model.

Explore the system design: Select a component to see its task, incoming and outgoing data, and role in the use case. The guided walkthrough explains each phase. Then use Yes, Uncertain and No to represent your specific system variant.

22 review first15 clarify8 baseline threats

InputProcessingResultImpact Externalaccess 04 Input 31 Third-partycontent 10 AI model 40 Providerservice 11 Documentcollection 01 Memory 01 Training andfeedback 02 OtherAI agents 00 Tools andinterfaces 02 Result 122 Business appor equipment 00 person 10 Affectedperson 00 AI serviceswithout approval 01 Externalaccess 04 Input 31 Third-partycontent 10 AI model 40 Providerservice 11 Documentcollection 01 Memory 01 Training andfeedback 02 OtherAI agents 00 Tools andinterfaces 02 Result 122 Business appor equipment 00 person 10 Affectedperson 00 AI serviceswithout approval 01
  • Present
  • Depends on your system
  • Absent
  • Review first
  • To clarify
Architecture as a list

How it differs from similar use cases

Data and information

Input data and information sources

Staff paste third-party emails, websites and documents for processing. Texts to be rewritten or summarised often mention customers, citizens or colleagues.

Processing and operating environment

The language model generally comes from another provider, even when hosted internally.

Storage and further use

Some assistants search uploaded files or the internet. Some assistants remember information from earlier conversations and consult it later.

Results and data sharing

Some assistants initiate web searches or call other services themselves. If a cloud service hosts the language model, inputs leave the organisation's environment.

Using a provider does not automatically determine where processing takes place. Storage location, subcontractors, deletion periods and use for training must be checked for the specific deployment.

Assess risks and determine measures

From understanding the system to analysing risks

  1. Define the system boundary

    Record data, participants, interfaces and permissions. Resolve uncertain characteristics in the model.

  2. Investigate threats

    Relate review candidates to actual data flows: what could happen, and who would be affected?

  3. Assess risk

    Assess impacts and likelihood in the deployment context. The number of review candidates is not a risk score.

  4. Provide evidence of measures

    Select appropriate controls, assign responsibilities and define effectiveness checks. Document residual risks and approvals.

Key threats in the application context

  • 4 particularly important
  • 18 typical
  • 15 depending on the system
  • 8 additional threats for every AI system

A total of 45 threats from the threat catalogue may be relevant to this use case.

Priority review points

  1. Prompt Injection: Indirect

    A staff member summarises an incoming email containing hidden instructions. The assistant follows them and returns a distorted summary or misleading recommendation.

  2. Sensitive Information Disclosure

    Staff paste personnel records, contracts or source code for processing. If the service stores the inputs or sends them to its provider, confidential content leaves the organisation.

  3. Overreliance

    Drafts sound confident and read smoothly. People who accept them under time pressure may stop checking content and sources.

  4. Factual Inconsistencies (Hallucinations)

    A draft cites a nonexistent provision, judgment or figure. If someone uses the text, the error enters correspondence to customers or citizens.

18 further review candidates in the typical design

Other relevant threats

  1. Backdoor ML Model

    Training data or an adopted base model can introduce hidden triggers for misbehaviour.

    Listed because of this characteristic

    • Uses another provider's model

  2. Sleepy Agent (Time/Event-Triggered Hidden Instructions)

    An instruction-following model can change behaviour at a particular time or event.

    Listed because of this characteristic

    • Performs natural language tasks

  3. Prompt Injection: Direct

    Users can override the system's working instructions through their inputs.

    Listed because of this characteristic

    • Performs natural language tasks

  4. Jailbreaks

    People requesting generated content may try to bypass intended limits.

    Listed because of this characteristic

    • Performs natural language tasks

  5. Meta Prompt Extraction

    Carefully crafted queries can reveal system instructions, which must therefore contain no secrets.

    Listed because of this characteristic

    • Performs natural language tasks

  6. Model Denial of Service

    Complex or high-volume inputs can exhaust model computing resources.

    Listed because of this characteristic

    • Generates its own text

  7. Privacy Attacks

    Targeted queries can recover some personal or protected training data from the model.

    Listed because of these characteristics

    • Processes personal data
    • Processes confidential data

  8. Exfiltration from ML Application

    Protected data can leak through tools, network connections, storage or insiders.

    Listed because of this characteristic

    • Processes confidential data

  9. Hate Speech and Discrimination

    Open-ended generated content can contain discriminatory or hateful statements.

    Listed because of these characteristics

    • Generates its own text
    • Generates content from open-ended requests

  10. Profanity

    Freely generated answers can use an inappropriate tone.

    Listed because of these characteristics

    • Generates its own text
    • Generates content from open-ended requests

  11. Violence / Unsafe Actions

    Open-ended content can depict violence or assist dangerous actions.

    Listed because of these characteristics

    • Generates its own text
    • Generates content from open-ended requests

  12. Controversial Topics

    Answers on controversial topics may require separate editorial review.

    Listed because of these characteristics

    • Generates its own text
    • Generates content from open-ended requests

  13. Illegal Activities

    Open-ended content can facilitate unlawful acts. The required scrutiny depends on purpose and users.

    Listed because of these characteristics

    • Generates its own text
    • Generates content from open-ended requests

  14. Harassment

    Open-ended content can be used for harassment.

    Listed because of these characteristics

    • Generates its own text
    • Generates content from open-ended requests

  15. Unethical Actions

    Generated suggestions can cross ethical boundaries in the specific context.

    Listed because of these characteristics

    • Generates its own text
    • Generates content from open-ended requests

  16. Social Engineering

    Generated images, voices and texts can deceive or impersonate people.

    Listed because of these characteristics

    • Generates its own text
    • Generates content from open-ended requests

  17. Fraud

    Generated content can facilitate fraud, for example through forged messages or documents.

    Listed because of these characteristics

    • Generates its own text
    • Generates content from open-ended requests

  18. Disinformation

    Synthetic content can be presented as authentic and spread misleadingly.

    Listed because of these characteristics

    • Generates its own text
    • Generates content from open-ended requests

Up to 15 further threats may apply, depending on your system design. Nine threats need to be reviewed for every AI system. One of these is particularly important here and is therefore listed above.

System features and open questions

Each Yes adds further threats to your checklist, up to 15 in total. If you cannot yet answer a question, include its threats in your review.

How many threats a Yes adds
  • Added by this Yes
  • Also requires a Yes to another question

How your system is built

  1. Calls other programs or interfaces2 threats
  2. Uses operational feedback2 threats
  3. Learns from your own data1 threat, plus 1 only in combination with other characteristics
  4. Generates images, audio or videos1 threat
  5. Searches its own document collection1 threat
  6. Formulates answers from retrieved documents1 threat
  7. Remembers content beyond a single conversation1 threat
  8. Learns from examples with correct answers1 threat
  9. Processes text, images and speech together1 threat
  10. Generates code or queries1 threat
  11. Executes programs or commands1 threat

How you operate the system

  1. Accepts queries directly through an interface4 threats
  2. Accessible over a network1 threat, plus 1 only in combination with other characteristics
  3. Incurs costs with each query1 threat
  4. Shares computing resources with other customers or applications1 threat
  5. Used alongside unapproved AI services1 threat

How your system is built

Does the system generate or edit images, sounds, voices or videos?

Some assistants also generate images on request; text-only assistants do not.

If yes, add

Does the system remember content from earlier conversations or tasks and reuse it later?

Some assistants remember information from earlier conversations and consult it later.

If yes, add

Does the system call other programs, interfaces or search services itself, even if only to read?

Some assistants initiate web searches or call other services themselves.

If yes, add

Does your organisation train or fine-tune the model itself using its own data?

Most deployers use the language model unchanged; some fine-tune it on their own texts.

If yes, add

Does the system process several input types together, such as text with images or speech?

Many assistants accept images, tables or speech as well as text.

If yes, add

Does the system generate code, scripts or database queries for further use?

Staff also request spreadsheet formulas, scripts or database queries.

If yes, add

Does the system execute programs, scripts or commands itself?

Some assistants execute code themselves, for example to analyse spreadsheets.

If yes, add

How you operate the system

These questions arise for almost every AI system.

Do you pay per query or unit of computing time, so costs increase with use?

With a cloud service, every request costs money.

If yes, add

Do staff also use AI services for this task that your organisation has not approved?

Without an approved assistant, staff often use public services with private accounts.

If yes, add

Controls and implementation

Measures for the priority review areas are expanded. Each linked measure explains how to implement it.

Review priorityPrompt Injection: Indirect3 measures
Review prioritySensitive Information Disclosure3 measures
Review priorityOverreliance3 measures
Review priorityFactual Inconsistencies (Hallucinations)3 measures
Backdoor ML Model2 measures
Sleepy Agent (Time/Event-Triggered Hidden Instructions)3 measures
Prompt Injection: Direct3 measures
Jailbreaks2 measures
Meta Prompt Extraction2 measures
Model Denial of Service3 measures
Privacy Attacks3 measures
Exfiltration from ML Application3 measures
Hate Speech and Discrimination2 measures
Profanity2 measures
Violence / Unsafe Actions2 measures
Controversial Topics3 measures
Illegal Activities3 measures
Harassment3 measures
Unethical Actions3 measures
Social Engineering3 measures
Fraud3 measures
Disinformation2 measures

Controls for the remaining threats are available on their pages in the threat catalogue.

AI Act provisions

When AI Act provisions apply
  1. Art. 4: AI literacy

    For every AI system Applies since 2 February 2025.

  2. Art. 53 para. 1 point b, Annex XII: Embedded general-purpose model

    Depending on use Review trigger since 2 August 2025.

  3. Art. 50 para. 2: Marking artificially generated content

    Depending on use Review trigger since 2 August 2026.

  4. Art. 50 para. 4: Labelling deepfakes and published texts

    Depending on use Review trigger since 2 August 2026.

Applies to every AI system

Art. 4: AI literacy

Applies if staff or appointed representatives operate or use AI systems.

Applies since 2 February 2025.

Obligations and details
Who bears the obligations
Provider, Deployer, Product manufacturer, Provider of a general-purpose AI model

Detailed obligations

May apply in particular uses

Art. 50 para. 2: Marking artificially generated content

Applies if your organisation develops the assistant itself and provides it under its own name.

Applies since 2 August 2026.

Obligations and details
The general condition is
Applies if the system generates synthetic text, images, audio or video.
Exception
Exceptions include ordinary editing assistance that does not substantially alter inputs or their meaning.
Points to consider
Systems marketed before 2 August 2026 must be updated by 2 December 2026.
Who bears the obligations
Provider, Product manufacturer, Provider of a general-purpose AI model

Detailed obligations

Art. 50 para. 4: Labelling deepfakes and published texts

Applies if texts produced with the assistant are published on matters of public interest without human editorial review and responsibility.

Applies since 2 August 2026.

Obligations and details
The general condition is
Applies if the system creates deepfakes or AI texts on matters of public interest are published without editorial responsibility.
Who bears the obligations
Deployer

Detailed obligations

Art. 53 para. 1 point b, Annex XII: Embedded general-purpose model

Applies if your organisation incorporates a large language model or another third-party general-purpose model into its own assistant.

Applies since 2 August 2025.

Obligations and details
The general condition is
Applies if another provider's general-purpose AI model is incorporated, such as a large language model.
Points to consider
The model provider bears its obligations. It must give you the Annex XII information; use it for your own documentation.
Who bears the obligations
Provider of a general-purpose AI model

Detailed obligations

Generally does not apply

Art. 50 para. 1: Disclosure of AI interaction

Staff knowingly invoke the assistant, making it obvious that they are using an AI system.

Obligations and details
The general condition is
Applies if people directly write or speak with the system and it is not obvious that they are interacting with AI.
Who bears the obligations
Provider, Product manufacturer

Detailed obligations

Classify your system with the quick checker All obligations in the obligations compass

Cross-cutting threats

Review these threats for every AI system, whatever its purpose.

Why these threats always belong on the checklist

Atypical system features

View 7 further questions

If any of these features does apply to your system, add the listed threats.

Sources

Related use cases

Also in the field of Office work and knowledge management

Updated: 30 September 2026. Review checklist rules version 0.3.2, threat catalogue version 2026.10.24.2. How these pages are structured