Input data and information sources
Reference photographs, voice recordings or images often come from third parties. References or outputs showing real people process their likenesses or voices.
Image, video and voice generation
Also known as: Generative media (image, video, audio)
The system generates or edits images, videos and audio from a short description, for example for advertising and communication.
Understand the use case
What defines this use case?
Descriptions or references produce new images, scenes or voices that can be hard to distinguish from real recordings. Major brands use these tools to produce advertising films and images.
This page describes a typical system design. Actual data flows and permissions depend on your implementation.
- System design
- Media generation
- Ability to act
- Provides results for subsequent use
- Fields of use
- marketing, beverage industry, payment services
- Also known as
- text-to-image, synthetic media, AI video, deepfake, voice clone
How it works, step by step
System design and data flows
Input data
A person describes the desired scene and optionally attaches photographs, videos or voice recordings as references.
Reference photographs, voice recordings or images often come from third parties. Editing existing images or videos from a description processes image and text together.
Processing
A generative model creates images, scenes or sounds or edits the reference.
A generative model processes the description and any reference material to create new image, audio or video content. Generation can involve multiple variants and editing steps. The result is generated material and does not establish that a depicted scene actually occurred.
Results
Finished images, videos or audio often convincingly resemble real recordings.
A generated result is newly worded or created. To assess it, the instruction and context on which it is based must remain identifiable.
Use and impact
Outputs appear in advertising, social media and communication and shape what people believe is real.
The model supplies information for subsequent use. Selection, adoption or a professional decision in the following workflow is a separate step and should not automatically be attributed to the model.
Explore the system design: Select a component to see its task, incoming and outgoing data, and role in the use case. The guided walkthrough explains each phase. Then use Yes, Uncertain and No to represent your specific system variant.
16 review first16 clarify9 baseline threats
- Present
- Depends on your system
- Absent
- Review first
- To clarify
How to use the model
The diagram shows the typical architecture. Select a component to see what happens there and adapt its features to your system. Enable “Analysis guidance” to see which threats need reviewing for this architecture. The page's checklist continues to describe the typical case.
Changes from the typical case
None yet. Each change will appear here.
Where most threats arise
- 11 review first, 3 clarify
- 5 clarify
- 3 review first
- 1 review first, 2 clarify
In this system design: 16 to review first, 16 to clarify, 9 baseline threats.
Architecture as a list
External access: Depends on your system
To clarify
Input: Part of every AI system
To clarify
Baseline for every AI system
Third-party content: Present
Review first
AI model: Part of every AI system
Review first
- Sleepy Agent (Time/Event-Triggered Hidden Instructions)
- Prompt Injection: Direct
- Meta Prompt Extraction
Baseline for every AI system
Provider service: Present
Review first
To clarify
Training and feedback: Depends on your system
To clarify
Result: Part of every AI system
Review first
- Jailbreaks Review priority
- Sexual Content Review priority
- Disinformation Review priority
- Model Denial of Service
- Hate Speech and Discrimination
- Violence / Unsafe Actions
- Illegal Activities
- Harassment
- Unethical Actions
- Social Engineering
- Fraud
To clarify
Baseline for every AI system
person: Part of every AI system
Baseline for every AI system
Unapproved AI services: Depends on your system
To clarify
Supply chain and technical foundations: 5 baseline threats applying to every AI system
Baseline for every AI system
How it differs from similar use cases
Customer service chatbots and voicebots
Converses with customers, including using a synthetic voice
Generative AI assistant for staff
Primarily generates text and only occasionally an image
Data and information
Processing and operating environment
Generators come from providers such as Midjourney, OpenAI or Adobe.
Storage and further use
Some deployers fine-tune generators on their own images, for example for consistent branding.
Results and data sharing
Cloud services send descriptions and references outside the organisation's environment.
Using a provider does not automatically determine where processing takes place. Storage location, subcontractors, deletion periods and use for training must be checked for the specific deployment.
Assess risks and determine measures
From understanding the system to analysing risks
- Define the system boundary
Record data, participants, interfaces and permissions. Resolve uncertain characteristics in the model.
- Investigate threats
Relate review candidates to actual data flows: what could happen, and who would be affected?
- Assess risk
Assess impacts and likelihood in the deployment context. The number of review candidates is not a risk score.
- Provide evidence of measures
Select appropriate controls, assign responsibilities and define effectiveness checks. Document residual risks and approvals.
Key threats in the application context
- 3 particularly important
- 13 typical
- 16 depending on the system
- 9 for every AI system
A total of 41 threats from the threat catalogue may be relevant to this use case.
Priority review points
Someone uploads a colleague's photograph and requests a revealing image. Without safeguards, this produces a non-consensual intimate depiction that seriously harms the person.
An advertising video shows a realistic scene involving a well-known person that never happened. Without AI disclosure, viewers believe it is real and share it further.
Carefully rephrased prompts bypass generator safeguards. It then produces violent depictions or protected brands attributed to the deployer.
13 further review candidates in the typical design
Other relevant threats
Training data or an adopted base model can introduce hidden triggers for misbehaviour.
Listed because of this characteristic
- Uses another provider's model
Sleepy Agent (Time/Event-Triggered Hidden Instructions)
An instruction-following model can change behaviour at a particular time or event.
Listed because of this characteristic
- Performs natural language tasks
Users can override the system's working instructions through their inputs.
Listed because of this characteristic
- Performs natural language tasks
A model can follow instructions in a document or tool result as if they were its own working instructions.
Listed because of these characteristics
- Performs natural language tasks
- Processes content outsiders can write
Carefully crafted queries can reveal system instructions, which must therefore contain no secrets.
Listed because of this characteristic
- Performs natural language tasks
Complex or high-volume inputs can exhaust model computing resources.
Listed because of this characteristic
- Generates images, audio or videos
Hate Speech and Discrimination
Open-ended generated content can contain discriminatory or hateful statements.
Listed because of these characteristics
- Generates images, audio or videos
- Generates content from open-ended requests
Open-ended content can depict violence or assist dangerous actions.
Listed because of these characteristics
- Generates images, audio or videos
- Generates content from open-ended requests
Open-ended content can facilitate unlawful acts. The required scrutiny depends on purpose and users.
Listed because of these characteristics
- Generates images, audio or videos
- Generates content from open-ended requests
Open-ended content can be used for harassment.
Listed because of these characteristics
- Generates images, audio or videos
- Generates content from open-ended requests
Generated suggestions can cross ethical boundaries in the specific context.
Listed because of these characteristics
- Generates images, audio or videos
- Generates content from open-ended requests
Generated images, voices and texts can deceive or impersonate people.
Listed because of this characteristic
- Generates images, audio or videos
Generated content can facilitate fraud, for example through forged messages or documents.
Listed because of this characteristic
- Generates images, audio or videos
Up to 16 further threats may apply, depending on your system design. Nine threats need to be reviewed for every AI system.
System features and open questions
Each Yes adds further threats to your checklist, up to 16 in total. If you cannot yet answer a question, include its threats in your review.
- Added by this Yes
- Also requires a Yes to another question
How your system is built
- Generates its own text3 threats, plus 1 only in combination with other characteristics
- Processes confidential data3 threats
- Processes personal data2 threats
- Uses operational feedback2 threats
- Learns from your own data1 threat, plus 1 only in combination with other characteristics
- Recognises or compares people by physical features1 threat
- Learns from examples with correct answers1 threat
- Processes text, images and speech together1 threat
How you operate the system
- Available to people outside the organisation5 threats
- Accepts queries directly through an interface3 threats, plus 1 only in combination with other characteristics
- Accessible over a network1 threat, plus 1 only in combination with other characteristics
- Incurs costs with each query1 threat
- Passes content to services outside your environment1 threat
- Shares computing resources with other customers or applications1 threat
- Used alongside unapproved AI services1 threat
How your system is built
Does the system generate its own text, such as answers, drafts or summaries?
Some tools also write scripts, advertising copy or subtitles.
If yes, add
Does the system process information about identifiable people, including in inputs, outputs or logs?
References or outputs showing real people process their likenesses or voices.
If yes, add
Does the system process confidential information such as trade secrets, health or financial data?
If yes, add
Does the system use faces, voices, fingerprints or other physical features to recognise or compare people?
Replicating a specific person's voice or face uses their physical characteristics.
If yes, add
Does your organisation train or fine-tune the model itself using its own data?
Some deployers fine-tune generators on their own images, for example for consistent branding.
If yes, add
Is the model trained on examples to which someone has assigned the correct answer?
If yes, add
Does operational feedback or output later feed into analysis or training?
If yes, add
Does the system process several input types together, such as text with images or speech?
Editing existing images or videos from a description processes image and text together.
If yes, add
How you operate the system
These questions arise for almost every AI system.
Can people or programs query the model directly, for example through an interface?
If yes, add
Is the system accessible over a network or dependent on network services?
If yes, add
Do you pay per query or unit of computing time, so costs increase with use?
Cloud services charge for each generated image or video.
If yes, add
Can content leave your environment, for example to a cloud provider, web search or email?
Cloud services send descriptions and references outside the organisation's environment.
If yes, add
Does the system share servers or computing resources with other customers or applications?
If yes, add
Can people outside your organisation use the system, including without login?
An organisation offering the generator to customers also allows outside users.
If yes, add
Do staff also use AI services for this task that your organisation has not approved?
Agencies and staff often use freely available generators with their own accounts.
If yes, add
Controls and implementation
Measures for the priority review areas are expanded. Each linked measure explains how to implement it.
Review prioritySexual Content3 measures
Review priorityDisinformation2 measures
Review priorityJailbreaks2 measures
Backdoor ML Model2 measures
Sleepy Agent (Time/Event-Triggered Hidden Instructions)3 measures
Prompt Injection: Direct3 measures
Prompt Injection: Indirect3 measures
Meta Prompt Extraction2 measures
Model Denial of Service3 measures
Hate Speech and Discrimination2 measures
Violence / Unsafe Actions2 measures
Illegal Activities3 measures
Harassment3 measures
Unethical Actions3 measures
Social Engineering3 measures
Controls for the remaining threats are available on their pages in the threat catalogue.
AI Act provisions
-
Art. 4: AI literacy
For every AI system Applies since 2 February 2025.
-
Art. 53 para. 1 point b, Annex XII: Embedded general-purpose model
Depending on use Review trigger since 2 August 2025.
-
Art. 50 para. 2: Marking artificially generated content
In typical use Review trigger since 2 August 2026.
-
Art. 50 para. 4: Labelling deepfakes and published texts
In typical use Review trigger since 2 August 2026.
-
Art. 5 para. 1 Upara. 1 point ba, Art. 5 para. 1a, 1b: Non-consensual intimate depictions
In typical use Review trigger from 2 December 2026.
-
Art. 5 para. 1 Upara. 1 point bb: Child sexual abuse material
In typical use Review trigger from 2 December 2026.
Applies to every AI system
Art. 4: AI literacy
Applies if staff or appointed representatives operate or use AI systems.
Applies since 2 February 2025.
Obligations and details for Art. 4: AI literacy
- Who bears the obligations
- Provider, Deployer, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
May apply in typical use
Art. 5 para. 1 Upara. 1 point ba, Art. 5 para. 1a, 1b: Non-consensual intimate depictions
Applies if the tool can generate or manipulate realistic images, videos or audio of identifiable people, for example from uploaded photographs.
Applies from 2 December 2026.
Obligations and details for Art. 5 para. 1 Upara. 1 point ba, Art. 5 para. 1a, 1b: Non-consensual intimate depictions
- The general condition is
- Applies if the system can generate or manipulate realistic images, videos or audio of identifiable people.
- Points to consider
- For deployers, the prohibition applies only when they use the system for this specific purpose.
- Who bears the obligations
- Provider, Deployer, Importer, Distributor, Authorised representative, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
Art. 5 para. 1 Upara. 1 point bb: Child sexual abuse material
Applies if it can generate or manipulate realistic images, videos or audio without effectively preventing sexual depictions of children.
Applies from 2 December 2026.
Obligations and details for Art. 5 para. 1 Upara. 1 point bb: Child sexual abuse material
- The general condition is
- Applies if the system can generate or manipulate realistic images, videos or audio.
- Who bears the obligations
- Provider, Deployer, Importer, Distributor, Authorised representative, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
Art. 50 para. 2: Marking artificially generated content
Applies if your organisation provides the generator itself or under its own name. Generated images, videos and audio must then carry machine-readable marking.
Applies since 2 August 2026.
Obligations and details for Art. 50 para. 2: Marking artificially generated content
- The general condition is
- Applies if the system generates synthetic text, images, audio or video.
- Exception
- Exceptions include ordinary editing assistance that does not substantially alter inputs or their meaning.
- Points to consider
- Systems marketed before 2 August 2026 must be updated by 2 December 2026.
- Who bears the obligations
- Provider, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
Art. 50 para. 4: Labelling deepfakes and published texts
Applies if your organisation publishes images, videos or audio that deceptively resemble real people, places or events.
Applies since 2 August 2026.
Obligations and details for Art. 50 para. 4: Labelling deepfakes and published texts
- The general condition is
- Applies if the system creates deepfakes or AI texts on matters of public interest are published without editorial responsibility.
- Who bears the obligations
- Deployer
Detailed obligations
May apply in particular uses
Art. 53 para. 1 point b, Annex XII: Embedded general-purpose model
Applies if your organisation incorporates a large language model or another third-party general-purpose model into its own image or video tool.
Applies since 2 August 2025.
Obligations and details for Art. 53 para. 1 point b, Annex XII: Embedded general-purpose model
- The general condition is
- Applies if another provider's general-purpose AI model is incorporated, such as a large language model.
- Points to consider
- The model provider bears its obligations. It must give you the Annex XII information; use it for your own documentation.
- Who bears the obligations
- Provider of a general-purpose AI model
Detailed obligations
Classify your system with the quick checker All obligations in the obligations compass
Cross-cutting threats
Review these threats for every AI system, whatever its purpose.
Atypical system features
View 14 further questions
If any of these features does apply to your system, add the listed threats.
Does the system search its own collection of documents or records for relevant content?
If yes, add
Does the system formulate answers from retrieved documents?
If yes, add
Does the system use a knowledge graph storing concepts and their relationships?
If yes, add
Does the system remember content from earlier conversations or tasks and reuse it later?
If yes, add
Does the system call other programs, interfaces or search services itself, even if only to read?
If yes, add
Can the system change things itself, such as storing data, sending messages, initiating transactions or granting permissions?
If yes, add
Does the system determine its next steps itself based on earlier results?
If yes, add
Does the system delegate tasks to other AI agents or incorporate their results?
If yes, add
Is the system connected to tools or data sources through the Model Context Protocol (MCP)?
If yes, add
Do documents from several customers, departments or permission groups share one search store?
If yes, add
Does the system directly move or control something physical, such as motors, valves, vehicles or temperatures?
If yes, add
Does the system generate code, scripts or database queries for further use?
If yes, add
Does the system execute programs, scripts or commands itself?
If yes, add
Is the system intended to maximise a specified metric, such as revenue, utilisation or clicks?
If yes, add
Sources
- Statistics
Federal Statistical Office (Destatis), AI-using enterprises by technology type, 2025
Among AI-using companies, 52 per cent use image, video or audio generation technologies.
- Report of a deployment
Marketing Week, Coca-Cola AI version of Holidays Are Coming
Coca-Cola commissioned an advertisement made with generative AI.
- Report of a deployment
Marketing Dive, Klarna, 29 May 2024 According to the deployer
Klarna states that it creates and edits advertising images using image generators.
- Authority, court or parliament
Federal Office for Information Security (BSI), Generative AI models, 2025
The BSI treats image and video generators as a distinct model class.
- Authority, court or parliament
European Commission, Questions and answers on Article 50 AI Act transparency obligations
The Commission explains labelling of synthetic content and deepfakes. Systems marketed before 2 August 2026 have until 2 December 2026.
Related use cases
Also in the field of Customer service, marketing and platforms
- Customer service chatbots and voicebotsA bot independently handles customers' or citizens' requests in chat or by phone and passes difficult cases to people.
- Recommender systemThe system uses a person's behaviour and profile to select products, content or search results and determine their order.
- Dynamic pricingOnline retailers and platforms automatically adjust prices to demand, costs, capacity and competitors' prices.
- Personalised online advertising (behavioural targeting)The system uses a person's behaviour and profile to select which advertisements they see.
- Automated content moderationPlatforms automatically identify unlawful content or breaches of their rules and remove, restrict or refer posts for human review.
Updated: 30 September 2026. Review checklist rules version 0.3.2, threat catalogue version 2026.10.24.2. How these pages are structured