Input data and information sources
All reviewed posts come from users, including those deliberately seeking to bypass moderation. Posts, accounts and reports are linked to specific people.
Automated content moderation
Platforms automatically identify unlawful content or breaches of their rules and remove, restrict or refer posts for human review.
Understand the use case
What defines this use case?
Models check text, images and videos against laws or terms of service and determine what disappears immediately and what a person reviews. Major platforms use this approach; a German federal agency uses it to detect offensive language on its intranet.
This page describes a typical system design. Actual data flows and permissions depend on your implementation.
- System design
- Analysis and processing
- Ability to act
- Can act within applications
- Fields of use
- online platforms, public administration
- Also known as
- content moderation, toxicity detection, comment filter
How it works, step by step
System design and data flows
Input data
Posts, comments, images and videos are checked on upload or after a report.
All reviewed posts come from users, including those deliberately seeking to bypass moderation. The system reviews text, images and videos, often including text within images.
Processing
The system matches them against known prohibited content; models assess whether they violate laws or terms of service.
The model detects characteristics in text, images, audio or video and assigns them to the platform's moderation categories. Rules and thresholds then determine whether content is flagged, held for review or restricted. Detecting a characteristic and making a moderation decision are separate steps.
Results
The post is removed, restricted, referred to a person or left in place.
The result describes the defined model task. A match, category or forecast acquires its professional significance only in conjunction with the underlying data and intended use.
Use and impact
The decision determines what remains publicly visible and can suspend accounts or reduce earnings.
Write or execution permissions allow the system to change a state within an application. Model responses, tool calls and changes actually executed must therefore be considered separately.
Explore the system design: Select a component to see its task, incoming and outgoing data, and role in the use case. The guided walkthrough explains each phase. Then use Yes, Uncertain and No to represent your specific system variant.
9 review first13 clarify8 baseline threats
- Present
- Depends on your system
- Absent
- Review first
- To clarify
How to use the model
The diagram shows the typical architecture. Select a component to see what happens there and adapt its features to your system. Enable “Analysis guidance” to see which threats need reviewing for this architecture. The page's checklist continues to describe the typical case.
Changes from the typical case
None yet. Each change will appear here.
Where most threats arise
- 5 review first
- 5 clarify
- 3 review first
- 3 clarify
In this system design: 9 to review first, 13 to clarify, 8 baseline threats.
Architecture as a list
External access: Depends on your system
To clarify
Input: Part of every AI system
Review first
- Adversarial Inputs Review priority
- Multimodal Attacks Review priority
- Model Denial of Service
- Privacy Attacks
- Sensitive Information Disclosure
Third-party content: Present
AI model: Part of every AI system
To clarify
- Sleepy Agent (Time/Event-Triggered Hidden Instructions)
- Prompt Injection: Direct
- Prompt Injection: Indirect
- Jailbreaks
- Meta Prompt Extraction
Baseline for every AI system
Provider service: Depends on your system
To clarify
Document collection: Depends on your system
To clarify
Training and feedback: Present
Review first
Result: Part of every AI system
Baseline for every AI system
Business application or equipment: Present
Review first
- Excessive Agency Review priority
person: Part of every AI system
Baseline for every AI system
Affected person: Present
Unapproved AI services: Depends on your system
To clarify
Supply chain and technical foundations: 5 baseline threats applying to every AI system
Baseline for every AI system
How it differs from similar use cases
Determines how prominently acceptable posts appear, rather than whether they are acceptable
Text analytics for feedback and consultation responses
Analyses feedback by theme in aggregate and removes no posts
Data and information
Processing and operating environment
Smaller deployers use ready-made third-party services such as Perspective API.
Storage and further use
To match known content, the system searches a collection of previously removed images and videos. Major platforms train their models on their own previously moderated posts.
Results and data sharing
The system removes posts, restricts their reach or suspends accounts, often without prior human review.
Using a provider does not automatically determine where processing takes place. Storage location, subcontractors, deletion periods and use for training must be checked for the specific deployment.
Assess risks and determine measures
From understanding the system to analysing risks
- Define the system boundary
Record data, participants, interfaces and permissions. Resolve uncertain characteristics in the model.
- Investigate threats
Relate review candidates to actual data flows: what could happen, and who would be affected?
- Assess risk
Assess impacts and likelihood in the deployment context. The number of review candidates is not a risk score.
- Provide evidence of measures
Select appropriate controls, assign responsibilities and define effectiveness checks. Document residual risks and approvals.
Key threats in the application context
- 4 particularly important
- 5 typical
- 13 depending on the system
- 8 additional threats for every AI system
A total of 30 threats from the threat catalogue may be relevant to this use case.
Also review: whether the system disadvantages groups
Check whether selection, ranking or assessment disadvantages people with protected characteristics, such as sex, age or origin.
Priority review points
Users replace letters in insults with digits or change a known image by a few pixels. The post passes review and remains online.
Targeted Poisoning / Label Poisoning
An organised group mass-reports a minority's legitimate posts as hate speech. If these reports enter training, the model learns to remove such posts itself.
A prohibited call appears as text in an image or only as speech in a video. Checking image, audio and text separately misses the violation.
The system can suspend accounts without human review. An error then affects many people at once before anyone notices.
5 further review candidates in the typical design
Other relevant threats
Your own training or operational feedback can introduce manipulated data into training datasets.
Listed because of these characteristics
- Learns from your own data
- Uses operational feedback
Training data or an adopted base model can introduce hidden triggers for misbehaviour.
Listed because of this characteristic
- Learns from your own data
Complex or high-volume inputs can exhaust model computing resources.
Listed because of this characteristic
- Processes text, images and speech together
Targeted queries can recover some personal or protected training data from the model.
Listed because of this characteristic
- Processes personal data
Sensitive Information Disclosure
Confidential content can appear in answers, supplied context or logs.
Listed because of this characteristic
- Processes personal data
Up to 13 further threats may apply, depending on your system design. Nine threats need to be reviewed for every AI system. One of these is particularly important here and is therefore listed above.
System features and open questions
Each Yes adds further threats to your checklist, up to 13 in total. If you cannot yet answer a question, include its threats in your review.
- Added by this Yes
How your system is built
- Performs natural language tasks5 threats
- Searches its own document collection1 threat
- Processes confidential data1 threat
How you operate the system
- Available to people outside the organisation4 threats
- Accepts queries directly through an interface3 threats
- Accessible over a network2 threats
- Incurs costs with each query1 threat
- Passes content to services outside your environment1 threat
- Shares computing resources with other customers or applications1 threat
- Used alongside unapproved AI services1 threat
How your system is built
Can users give the system natural language tasks that it then performs?
A language model reviewing posts against natural language guidelines may also follow instructions within a post.
If yes, add
Does the system search its own collection of documents or records for relevant content?
To match known content, the system searches a collection of previously removed images and videos.
If yes, add
Does the system process confidential information such as trade secrets, health or financial data?
Posts often concern authors' health, religion or political views.
If yes, add
How you operate the system
These questions arise for almost every AI system.
Can people or programs query the model directly, for example through an interface?
If yes, add
Is the system accessible over a network or dependent on network services?
If yes, add
Do you pay per query or unit of computing time, so costs increase with use?
If yes, add
Can content leave your environment, for example to a cloud provider, web search or email?
If yes, add
Does the system share servers or computing resources with other customers or applications?
If yes, add
Can people outside your organisation use the system, including without login?
If yes, add
Do staff also use AI services for this task that your organisation has not approved?
If yes, add
Controls and implementation
Measures for the priority review areas are expanded. Each linked measure explains how to implement it.
Review priorityAdversarial Inputs2 measures
Review priorityTargeted Poisoning / Label Poisoning3 measures
Review priorityMultimodal Attacks2 measures
Review priorityExcessive Agency3 measures
Training Data Poisoning2 measures
Backdoor ML Model2 measures
Model Denial of Service3 measures
Privacy Attacks3 measures
Sensitive Information Disclosure3 measures
Controls for the remaining threats are available on their pages in the threat catalogue.
AI Act provisions
Beyond AI literacy (Article 4), no other AI Act provision is relevant to this use case.
Applies to every AI system
Art. 4: AI literacy
Applies if staff or appointed representatives operate or use AI systems.
Applies since 2 February 2025.
Obligations and details for Art. 4: AI literacy
- Who bears the obligations
- Provider, Deployer, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
Other legal requirements
Articles 17 and 20 DSA
Platforms must explain moderation decisions, state whether they were automated and offer a complaint process whose decisions are not made solely by a machine.
Classify your system with the quick checker All obligations in the obligations compass
Cross-cutting threats
Review these threats for every AI system, whatever its purpose.
Atypical system features
View 14 further questions
If any of these features does apply to your system, add the listed threats.
Does the system generate its own text, such as answers, drafts or summaries?
If yes, add
Does the system generate or edit images, sounds, voices or videos?
If yes, add
Does the system create new content from open-ended requests that people read, see or hear directly? Translation, reading aloud and transcription do not count.
If yes, add
Does the system formulate answers from retrieved documents?
If yes, add
Does the system use a knowledge graph storing concepts and their relationships?
If yes, add
Does the system remember content from earlier conversations or tasks and reuse it later?
If yes, add
Does the system call other programs, interfaces or search services itself, even if only to read?
If yes, add
Does the system determine its next steps itself based on earlier results?
If yes, add
Does the system delegate tasks to other AI agents or incorporate their results?
If yes, add
Is the system connected to tools or data sources through the Model Context Protocol (MCP)?
If yes, add
Do documents from several customers, departments or permission groups share one search store?
If yes, add
Does the system generate code, scripts or database queries for further use?
If yes, add
Does the system execute programs, scripts or commands itself?
If yes, add
Is the system intended to maximise a specified metric, such as revenue, utilisation or clicks?
If yes, add
Sources
- Register or case collection
European Commission, DSA Transparency Database
The database records platforms' moderation decisions and shows the proportion made entirely automatically.
- Specialist study
The study analyses the transparency database legally and empirically, including automated moderation decisions.
- Specialist study
platform-governance.org, Analysis of the first Digital Services Act transparency reports, 2023
The analysis gives each platform's proportion of automated moderation.
- Report of a deployment
Federal Employment Agency, List of AI applications, 26 January 2026 (published via Table.Media)
The agency detects offensive language on its social intranet.
- Register or case collection
AI Incident Database, Incident 13 (Perspective API)
Perspective API, an offensive language detection service, produced biased assessments of certain groups' posts.
Related use cases
Also in the field of Customer service, marketing and platforms
- Customer service chatbots and voicebotsA bot independently handles customers' or citizens' requests in chat or by phone and passes difficult cases to people.
- Recommender systemThe system uses a person's behaviour and profile to select products, content or search results and determine their order.
- Dynamic pricingOnline retailers and platforms automatically adjust prices to demand, costs, capacity and competitors' prices.
- Personalised online advertising (behavioural targeting)The system uses a person's behaviour and profile to select which advertisements they see.
- Image, video and voice generationThe system generates or edits images, videos and audio from a short description, for example for advertising and communication.
Updated: 30 September 2026. Review checklist rules version 0.3.2, threat catalogue version 2026.10.24.2. How these pages are structured