Input data and information sources
Applicants supply the application information themselves. The system assesses specific individuals' or companies' applications and information.
Risk-based review selection for applications and benefits
Also known as: Risk scoring for benefit and grant applications
Authorities assess applications or payments for error or fraud risk and use this to select cases for closer review.
Understand the use case
What defines this use case?
When processing many applications, only some receive detailed checks; a model suggests which ones. UK and Dutch authorities use these models. In the Dutch childcare benefits scandal, thousands of families were wrongly suspected.
This page describes a typical system design. Actual data flows and permissions depend on your implementation.
- System design
- Analysis and processing
- Ability to act
- Action permissions depend on integration
- Fields of use
- social services, grant programmes
- Also known as
- risk selection, welfare fraud risk assessment, case selection
How it works, step by step
System design and data flows
Input data
The model receives application information, earlier applications and payments, and other business system data.
Applicants supply the application information themselves. The system assesses specific individuals' or companies' applications and information.
Processing
The model compares the application with earlier cases and calculates error or fraud risk.
The system processes information from applications and other approved records to estimate the need for review. A selection rule then prioritises cases for closer investigation. Selection for review, establishing the facts and deciding on a benefit are distinct procedural steps.
Results
Each application receives a risk score, with cases for closer review flagged.
The result describes the defined model task. A match, category or forecast acquires its professional significance only in conjunction with the underlying data and intended use.
Use and impact
Caseworkers review flagged cases, request evidence and decide on approval, reduction or recovery.
Integration determines whether the result is simply displayed, adopted after approval or written directly into a business application. These variants differ in their data flows and permissions.
Explore the system design: Select a component to see its task, incoming and outgoing data, and role in the use case. The guided walkthrough explains each phase. Then use Yes, Uncertain and No to represent your specific system variant.
8 review first9 clarify7 baseline threats
- Present
- Depends on your system
- Absent
- Review first
- To clarify
How to use the model
The diagram shows the typical architecture. Select a component to see what happens there and adapt its features to your system. Enable “Analysis guidance” to see which threats need reviewing for this architecture. The page's checklist continues to describe the typical case.
Changes from the typical case
None yet. Each change will appear here.
Where most threats arise
- 4 clarify
- 3 review first
- 3 review first
- 2 clarify
In this system design: 8 to review first, 9 to clarify, 7 baseline threats.
Architecture as a list
External access: Depends on your system
To clarify
Input: Part of every AI system
Review first
Baseline for every AI system
Third-party content: Present
AI model: Part of every AI system
To clarify
Baseline for every AI system
Provider service: Depends on your system
To clarify
Training and feedback: Present
Review first
Result: Part of every AI system
Review first
- Insecure Output Handling Review priority
Business application or equipment: Depends on your system
To clarify
person: Part of every AI system
Review first
- Overreliance Review priority
Affected person: Present
Unapproved AI services: Depends on your system
To clarify
Supply chain and technical foundations: 5 baseline threats applying to every AI system
Baseline for every AI system
How it differs from similar use cases
Assesses payments in a bank's or payment provider's transaction flow
Credit scoring and creditworthiness assessment
Assesses whether someone can repay a loan
Data and information
Processing and operating environment
The model processes the inputs. Whether it runs internally or with a service provider must be established for the specific implementation.
Storage and further use
Authorities train these models on their own past cases. Results of model-triggered checks become new training examples.
Results and data sharing
Depending on the authority, selection automatically stops a payment or only refers the case to staff.
Using a provider does not automatically determine where processing takes place. Storage location, subcontractors, deletion periods and use for training must be checked for the specific deployment.
Assess risks and determine measures
From understanding the system to analysing risks
- Define the system boundary
Record data, participants, interfaces and permissions. Resolve uncertain characteristics in the model.
- Investigate threats
Relate review candidates to actual data flows: what could happen, and who would be affected?
- Assess risk
Assess impacts and likelihood in the deployment context. The number of review candidates is not a risk score.
- Provide evidence of measures
Select appropriate controls, assign responsibilities and define effectiveness checks. Document residual risks and approvals.
Key threats in the application context
- 3 particularly important
- 5 typical
- 9 depending on the system
- 7 additional threats for every AI system
A total of 24 threats from the threat catalogue may be relevant to this use case.
Also review: whether the system disadvantages groups
Check whether selection, ranking or assessment disadvantages people with protected characteristics, such as sex, age or origin.
Priority review points
Reviewers can assume something is wrong with a selected case. Selection for review becomes suspicion, as in the Dutch childcare benefits scandal.
The cases reviewed and used for training are mainly those the model selected itself. This confirms earlier assumptions and increasingly targets groups already selected more often.
Selection passes into the business system and can stop payments. An incorrect risk score delays benefits for someone who depends on them.
5 further review candidates in the typical design
Other relevant threats
Targeted Poisoning / Label Poisoning
Falsified training labels can deliberately shift model outputs.
Listed because of these characteristics
- Learns from examples with correct answers
- Uses operational feedback
Training data or an adopted base model can introduce hidden triggers for misbehaviour.
Listed because of this characteristic
- Learns from your own data
Targeted queries can recover some personal or protected training data from the model.
Listed because of these characteristics
- Processes personal data
- Processes confidential data
Sensitive Information Disclosure
Confidential content can appear in answers, supplied context or logs.
Listed because of these characteristics
- Processes personal data
- Processes confidential data
Exfiltration from ML Application
Protected data can leak through tools, network connections, storage or insiders.
Listed because of this characteristic
- Processes confidential data
Up to 9 further threats may apply, depending on your system design. Nine threats need to be reviewed for every AI system. Two of these are particularly important here and are therefore listed above.
System features and open questions
Each Yes adds further threats to your checklist, up to 9 in total. If you cannot yet answer a question, include its threats in your review.
- Added by this Yes
How your system is built
- Changes data or triggers processes1 threat
- Optimises a specified metric1 threat
How you operate the system
How your system is built
Can the system change things itself, such as storing data, sending messages, initiating transactions or granting permissions?
Depending on the authority, selection automatically stops a payment or only refers the case to staff.
If yes, add
Is the system intended to maximise a specified metric, such as revenue, utilisation or clicks?
A model configured to maximise findings per check pursues an authority's objective.
If yes, add
How you operate the system
These questions arise for almost every AI system.
Can people or programs query the model directly, for example through an interface?
If yes, add
Is the system accessible over a network or dependent on network services?
If yes, add
Do you pay per query or unit of computing time, so costs increase with use?
If yes, add
Does the system share servers or computing resources with other customers or applications?
If yes, add
Do staff also use AI services for this task that your organisation has not approved?
If yes, add
Controls and implementation
Measures for the priority review areas are expanded. Each linked measure explains how to implement it.
Review priorityOverreliance3 measures
Review priorityTraining Data Poisoning2 measures
Review priorityInsecure Output Handling3 measures
Targeted Poisoning / Label Poisoning3 measures
Backdoor ML Model2 measures
Privacy Attacks3 measures
Sensitive Information Disclosure3 measures
Exfiltration from ML Application3 measures
Controls for the remaining threats are available on their pages in the threat catalogue.
AI Act provisions
-
Art. 4: AI literacy
For every AI system Applies since 2 February 2025.
-
Art. 5 para. 1 point c: Social scoring
Depending on use Review trigger since 2 February 2025.
-
Annex III no. 5 point a: Public assistance benefits
In typical use Review trigger from 2 December 2027.
Applies to every AI system
Art. 4: AI literacy
Applies if staff or appointed representatives operate or use AI systems.
Applies since 2 February 2025.
Obligations and details for Art. 4: AI literacy
- Who bears the obligations
- Provider, Deployer, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
May apply in typical use
Annex III no. 5 point a: Public assistance benefits
Applies if an authority uses the system to assess entitlement to public assistance benefits, for example by selecting applications for closer review, or to reduce or recover benefits.
Applies from 2 December 2027.
Obligations and details for Annex III no. 5 point a: Public assistance benefits
- The general condition is
- Applies if authorities or bodies acting for them use the system to assess eligibility for public benefits and services, or to grant, reduce or recover them.
- Who bears the obligations
- Provider, Deployer, Product manufacturer
Detailed obligations
May apply in particular uses
Art. 5 para. 1 point c: Social scoring
Applies if the system scores people over time by social behaviour or personal characteristics, causing disadvantages in other contexts or disproportionate disadvantages.
Applies since 2 February 2025.
Obligations and details for Art. 5 para. 1 point c: Social scoring
- The general condition is
- Applies if the system scores people over time by social behaviour or personal characteristics, causing harm in unrelated areas of life or disproportionate harm.
- Who bears the obligations
- Provider, Deployer, Importer, Distributor, Authorised representative, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
Classify your system with the quick checker All obligations in the obligations compass
Cross-cutting threats
Review these threats for every AI system, whatever its purpose.
Atypical system features
View 17 further questions
If any of these features does apply to your system, add the listed threats.
Does the system generate its own text, such as answers, drafts or summaries?
The system returns a risk score or selection, rather than text.
If yes, add
Does the system generate or edit images, sounds, voices or videos?
If yes, add
Does the system create new content from open-ended requests that people read, see or hear directly? Translation, reading aloud and transcription do not count.
If yes, add
Can users give the system natural language tasks that it then performs?
If yes, add
Does the system search its own collection of documents or records for relevant content?
If yes, add
Does the system formulate answers from retrieved documents?
If yes, add
Does the system use a knowledge graph storing concepts and their relationships?
If yes, add
Does the system remember content from earlier conversations or tasks and reuse it later?
If yes, add
Does the system call other programs, interfaces or search services itself, even if only to read?
If yes, add
Does the system determine its next steps itself based on earlier results?
If yes, add
Does the system delegate tasks to other AI agents or incorporate their results?
If yes, add
Is the system connected to tools or data sources through the Model Context Protocol (MCP)?
If yes, add
Do documents from several customers, departments or permission groups share one search store?
If yes, add
Can people outside your organisation use the system, including without login?
The model runs in the authority's background; applicants cannot access it.
If yes, add
Does the system process several input types together, such as text with images or speech?
If yes, add
Does the system generate code, scripts or database queries for further use?
If yes, add
Does the system execute programs, scripts or commands itself?
If yes, add
Sources
- Legislation, norm or standard
EU AI Act, Annex III, point 5(a)
The Act expressly lists AI systems assessing eligibility for public assistance benefits in Annex III.
- Register or case collection
UK Algorithmic Transparency Record, DWP Universal Credit Advances Model
The entry describes a classifier trained on examples with known outcomes.
- Authority, court or parliament
The Hague District Court, SyRI judgment, 5 February 2020
The Court found the legal basis for the Dutch fraud risk analysis system incompatible with Article 8 of the European Convention on Human Rights.
- Register or case collection
AI Incident Database, Incident 101 (Dutch childcare benefits)
The database documents thousands of families wrongly suspected of fraud.
- Register or case collection
Dutch Algorithm Register, Risicomodel omzetdaling NOW, Ministerie van SZW
The entry describes a model selecting NOW applications for closer review, decommissioned since January 2026.
Related use cases
Also in the field of Administration, justice and public safety
- Automated property valuation (AVM)A model estimates residential property market values from property characteristics and comparable sales.
- Police data analysis platformsPolice analysis platforms combine data from different sources and display links between people, incidents and cases as graphs, tables or maps.
- AI assistance in high-volume court proceedingsIn high-volume proceedings, the system organises pleadings, extracts each case's features and prepares draft decisions.
- Language and dialect recognition in asylum proceduresFrom a short speech sample, the system estimates which language and dialect an asylum seeker is likely to speak.
- Automatic number plate recognition (ANPR)Cameras capture vehicle number plates, and the system automatically checks them against registration or police search data.
- Intelligent video analytics in public spacesThe system detects specific events or objects in camera images and alerts a control centre without identifying people.
Often used together with
Updated: 30 September 2026. Review checklist rules version 0.3.2, threat catalogue version 2026.10.24.2. How these pages are structured