Input data and information sources
Ratings, third-party sellers' offers and clicking behaviour come from outsiders and can be deliberately manipulated. The system analyses individual usage behaviour and profiles.
Recommender system
The system uses a person's behaviour and profile to select products, content or search results and determine their order.
Understand the use case
What defines this use case?
The model learns from clicks, purchases and ratings what is likely to interest someone and ranks offerings accordingly. The OECD cites Amazon as a reference example, Netflix researches its recommendations' effects, and a Helsinki public library uses recommendations for books.
This page describes a typical system design. Actual data flows and permissions depend on your implementation.
- System design
- Analysis and processing
- Ability to act
- Provides results for subsequent use
- Fields of use
- online retail, streaming and media, libraries
- Also known as
- personalisation, product recommendations, ranking content and search results
How it works, step by step
System design and data flows
Input data
A person's clicks, purchases, ratings and profile details arrive continuously.
Ratings, third-party sellers' offers and clicking behaviour come from outsiders and can be deliberately manipulated. The system analyses individual usage behaviour and profiles.
Processing
A model estimates how likely the person is to respond to a product or content item and selects the best matches.
The model estimates which available content or products suit a person's current context. A ranking combines these estimates with platform rules, such as availability or diversity requirements. The recommendation therefore results from both model evaluation and selection rules.
Results
The system determines which products, content or search results appear and in what order.
The result describes the defined model task. A match, category or forecast acquires its professional significance only in conjunction with the underlying data and intended use.
Use and impact
The selection shapes what people buy, read or watch and generates data for further learning.
The model supplies information for subsequent use. Selection, adoption or a professional decision in the following workflow is a separate step and should not automatically be attributed to the model.
Explore the system design: Select a component to see its task, incoming and outgoing data, and role in the use case. The guided walkthrough explains each phase. Then use Yes, Uncertain and No to represent your specific system variant.
12 review first9 clarify9 baseline threats
- Present
- Depends on your system
- Absent
- Review first
- To clarify
How to use the model
The diagram shows the typical architecture. Select a component to see what happens there and adapt its features to your system. Enable “Analysis guidance” to see which threats need reviewing for this architecture. The page's checklist continues to describe the typical case.
Changes from the typical case
None yet. Each change will appear here.
Where most threats arise
- 1 review first, 5 clarify
- 5 review first
- 3 review first
- 2 review first, 1 clarify
In this system design: 12 to review first, 9 to clarify, 9 baseline threats.
Architecture as a list
External access: Present
Review first
Input: Part of every AI system
Review first
- Privacy Attacks Review priority
- Sensitive Information Disclosure
To clarify
Baseline for every AI system
Third-party content: Present
AI model: Part of every AI system
Review first
- Misalignment Review priority
To clarify
- Sleepy Agent (Time/Event-Triggered Hidden Instructions)
- Prompt Injection: Direct
- Prompt Injection: Indirect
- Jailbreaks
- Meta Prompt Extraction
Baseline for every AI system
Provider service: Depends on your system
To clarify
Document collection: Present
Review first
Training and feedback: Present
Review first
Result: Part of every AI system
Baseline for every AI system
person: Part of every AI system
Baseline for every AI system
Affected person: Depends on your system
Unapproved AI services: Depends on your system
To clarify
Supply chain and technical foundations: 5 baseline threats applying to every AI system
Baseline for every AI system
How it differs from similar use cases
Personalised online advertising (behavioural targeting)
Selects which paid advertisements a person sees
Sets an offer's price rather than its ranking
Decides whether a post is removed or its reach restricted
Data and information
Processing and operating environment
The model processes the inputs. Whether it runs internally or with a service provider must be established for the specific implementation.
Storage and further use
The system selects products or content from a catalogue to match the profile. The deployer trains the model on usage data from its own service.
Results and data sharing
The system determines which products, content or search results appear and in what order.
Using a provider does not automatically determine where processing takes place. Storage location, subcontractors, deletion periods and use for training must be checked for the specific deployment.
Assess risks and determine measures
From understanding the system to analysing risks
- Define the system boundary
Record data, participants, interfaces and permissions. Resolve uncertain characteristics in the model.
- Investigate threats
Relate review candidates to actual data flows: what could happen, and who would be affected?
- Assess risk
Assess impacts and likelihood in the deployment context. The number of review candidates is not a risk score.
- Provide evidence of measures
Select appropriate controls, assign responsibilities and define effectiveness checks. Document residual risks and approvals.
Key threats in the application context
- 3 particularly important
- 9 typical
- 9 depending on the system
- 9 for every AI system
A total of 30 threats from the threat catalogue may be relevant to this use case.
Priority review points
The system learns that provocative or sensational content increases time spent and recommends more of it. Engagement rises while users see increasingly harmful content.
A seller creates artificial interest through purchased ratings and automated clicks. The model learns from this and prominently recommends the products to many people.
The model infers pregnancy or illness from purchases and viewed content. Tailored recommendations reveal this to others sharing the device.
9 further review candidates in the typical design
Other relevant threats
Targeted Poisoning / Label Poisoning
Falsified training labels can deliberately shift model outputs.
Listed because of this characteristic
- Uses operational feedback
Training data or an adopted base model can introduce hidden triggers for misbehaviour.
Listed because of this characteristic
- Learns from your own data
Anyone able to query the model can replicate it through many queries. Internally trained weights can be stolen through network access.
Listed because of this characteristic
- Available to people outside the organisation
Complex or high-volume inputs can exhaust model computing resources.
Listed because of this characteristic
- Available to people outside the organisation
Paid or publicly triggered queries can be abused to consume computing resources and money.
Listed because of this characteristic
- Available to people outside the organisation
Sensitive Information Disclosure
Confidential content can appear in answers, supplied context or logs.
Listed because of this characteristic
- Processes personal data
A network-accessible application can be overwhelmed by mass requests.
Listed because of this characteristic
- Available to people outside the organisation
RAG-Specific Attacks (Document Poisoning)
Content entering the retrievable knowledge collection can influence answers and subsequent steps.
Listed because of this characteristic
- Searches its own document collection
Targeted queries can reveal the model's architecture and safeguards.
Listed because of this characteristic
- Available to people outside the organisation
Up to 9 further threats may apply, depending on your system design. Nine threats need to be reviewed for every AI system.
System features and open questions
Each Yes adds further threats to your checklist, up to 9 in total. If you cannot yet answer a question, include its threats in your review.
- Added by this Yes
How your system is built
- Performs natural language tasks5 threats
- Processes confidential data1 threat
- Processes text, images and speech together1 threat
How you operate the system
How your system is built
Can users give the system natural language tasks that it then performs?
Traditional recommender systems do not accept instructions; conversational services such as Obotti also process open-ended queries.
If yes, add
Does the system process confidential information such as trade secrets, health or financial data?
Purchases and viewed content can reveal health, religion or sexual orientation.
If yes, add
Does the system process several input types together, such as text with images or speech?
If yes, add
How you operate the system
These questions arise for almost every AI system.
Can content leave your environment, for example to a cloud provider, web search or email?
If yes, add
Does the system share servers or computing resources with other customers or applications?
If yes, add
Do staff also use AI services for this task that your organisation has not approved?
If yes, add
Controls and implementation
Measures for the priority review areas are expanded. Each linked measure explains how to implement it.
Review priorityMisalignment3 measures
Review priorityTraining Data Poisoning2 measures
Review priorityPrivacy Attacks3 measures
Targeted Poisoning / Label Poisoning3 measures
Backdoor ML Model2 measures
Model Theft3 measures
Model Denial of Service3 measures
Cost Harvesting / Repurposing3 measures
Sensitive Information Disclosure3 measures
Application Denial of Service3 measures
RAG-Specific Attacks (Document Poisoning)3 measures
Model Reconnaissance3 measures
Controls for the remaining threats are available on their pages in the threat catalogue.
AI Act provisions
-
Art. 4: AI literacy
For every AI system Applies since 2 February 2025.
-
Art. 5 para. 1 point a, b: Manipulation and exploitation of vulnerability
Depending on use Review trigger since 2 February 2025.
Applies to every AI system
Art. 4: AI literacy
Applies if staff or appointed representatives operate or use AI systems.
Applies since 2 February 2025.
Obligations and details for Art. 4: AI literacy
- Who bears the obligations
- Provider, Deployer, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
May apply in particular uses
Art. 5 para. 1 point a, b: Manipulation and exploitation of vulnerability
Applies if the system deliberately exploits vulnerabilities due to age, disability or economic circumstances in a way that risks significant harm.
Applies since 2 February 2025.
Obligations and details for Art. 5 para. 1 point a, b: Manipulation and exploitation of vulnerability
- The general condition is
- Applies if the system influences behaviour through subliminal, manipulative or deceptive means, or exploits vulnerability due to age, disability or social or economic circumstances, with a risk of significant harm.
- Who bears the obligations
- Provider, Deployer, Importer, Distributor, Authorised representative, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
Other legal requirements
Article 27 DSA
Online platforms must disclose in their terms and conditions the main parameters their recommender systems use to select and rank content.
Classify your system with the quick checker All obligations in the obligations compass
Cross-cutting threats
Review these threats for every AI system, whatever its purpose.
Atypical system features
View 14 further questions
If any of these features does apply to your system, add the listed threats.
Does the system generate its own text, such as answers, drafts or summaries?
If yes, add
Does the system generate or edit images, sounds, voices or videos?
If yes, add
Does the system create new content from open-ended requests that people read, see or hear directly? Translation, reading aloud and transcription do not count.
If yes, add
Does the system formulate answers from retrieved documents?
If yes, add
Does the system remember content from earlier conversations or tasks and reuse it later?
If yes, add
Does the system call other programs, interfaces or search services itself, even if only to read?
If yes, add
Can the system change things itself, such as storing data, sending messages, initiating transactions or granting permissions?
If yes, add
Does the system determine its next steps itself based on earlier results?
If yes, add
Does the system delegate tasks to other AI agents or incorporate their results?
If yes, add
Is the system connected to tools or data sources through the Model Context Protocol (MCP)?
If yes, add
Do documents from several customers, departments or permission groups share one search store?
If yes, add
Does the system directly move or control something physical, such as motors, valves, vehicles or temperatures?
If yes, add
Does the system generate code, scripts or database queries for further use?
If yes, add
Does the system execute programs, scripts or commands itself?
If yes, add
Sources
- Specialist study
OECD, Framework for the Classification of AI Systems, 2022, System 3
The OECD uses a recommender system as a reference example of personalisation.
- Legislation, norm or standard
Digital Services Act (DSA), Article 27 (dejure.org)
Online platforms must disclose the main parameters their recommender systems use to select and rank content.
- Register or case collection
Helsinki AI Register, Oodi’s book recommendation service Obotti
The city library operates a service recommending books according to readers' interests.
- Specialist study
The study measures how personalised recommendations affect viewing on Netflix.
Related use cases
Also in the field of Customer service, marketing and platforms
- Customer service chatbots and voicebotsA bot independently handles customers' or citizens' requests in chat or by phone and passes difficult cases to people.
- Dynamic pricingOnline retailers and platforms automatically adjust prices to demand, costs, capacity and competitors' prices.
- Personalised online advertising (behavioural targeting)The system uses a person's behaviour and profile to select which advertisements they see.
- Automated content moderationPlatforms automatically identify unlawful content or breaches of their rules and remove, restrict or refer posts for human review.
- Image, video and voice generationThe system generates or edits images, videos and audio from a short description, for example for advertising and communication.
Often used together with
Updated: 30 September 2026. Review checklist rules version 0.3.2, threat catalogue version 2026.10.24.2. How these pages are structured