Input data and information sources
Customer ratings and complaints that feed into performance assessments come from people outside the organisation. The system records who works where and when, and how quickly.
Algorithmic management
Software assigns tasks and shifts to workers, tracks what they do and evaluates their performance.
Understand the use case
What defines this use case?
The software takes over supervisory tasks: it allocates jobs, schedules shifts, measures speed and location, and uses these to generate ratings or bonuses. This is particularly familiar in delivery services. The Platform Work Directive (EU) 2024/2831 also regulates the use case under this name.
This page describes a typical system design. Actual data flows and permissions depend on your implementation.
- System design
- Analysis and processing
- Ability to act
- Can act within applications
- Fields of use
- platform work, delivery services
- Also known as
- automated task allocation, employee performance monitoring, shift and workforce scheduling
How it works, step by step
System design and data flows
Input data
The system receives jobs, availability, locations and measurements of speed, acceptance and customer ratings.
Customer ratings and complaints that feed into performance assessments come from people outside the organisation. The system records who works where and when, and how quickly.
Processing
The software allocates jobs and shifts according to its objectives and calculates performance figures for each person.
Processing combines work and operational data with a specified objective, such as workforce scheduling. This produces assignments, assessments or recommendations. Whether these are merely displayed or directly implemented as binding instructions depends on their integration into the work process.
Results
Workers receive assignments, shift schedules and performance figures, sometimes with bonuses or warnings.
The result describes the defined model task. A match, category or forecast acquires its professional significance only in conjunction with the underlying data and intended use.
Use and impact
Allocations and ratings determine workers' hours, income and career prospects.
Write or execution permissions allow the system to change a state within an application. Model responses, tool calls and changes actually executed must therefore be considered separately.
Explore the system design: Select a component to see its task, incoming and outgoing data, and role in the use case. The guided walkthrough explains each phase. Then use Yes, Uncertain and No to represent your specific system variant.
8 review first8 clarify8 baseline threats
- Present
- Depends on your system
- Absent
- Review first
- To clarify
How to use the model
The diagram shows the typical architecture. Select a component to see what happens there and adapt its features to your system. Enable “Analysis guidance” to see which threats need reviewing for this architecture. The page's checklist continues to describe the typical case.
Changes from the typical case
None yet. Each change will appear here.
Where most threats arise
- 4 clarify
- 3 review first
- 2 review first, 1 clarify
- 2 clarify
In this system design: 8 to review first, 8 to clarify, 8 baseline threats.
Architecture as a list
External access: Depends on your system
To clarify
Input: Part of every AI system
Review first
Baseline for every AI system
Third-party content: Depends on your system
AI model: Part of every AI system
Review first
- Misalignment Review priority
Baseline for every AI system
Provider service: Depends on your system
To clarify
Training and feedback: Present
Review first
- Training Data Poisoning Review priority
- Targeted Poisoning / Label Poisoning
To clarify
Result: Part of every AI system
Baseline for every AI system
Business application or equipment: Present
Review first
- Excessive Agency Review priority
person: Part of every AI system
Review first
- Overreliance Review priority
Affected person: Present
Unapproved AI services: Depends on your system
To clarify
Supply chain and technical foundations: 5 baseline threats applying to every AI system
Baseline for every AI system
How it differs from similar use cases
AI-assisted candidate screening
Sorts and evaluates applications before an employment relationship exists
Infers emotions from faces, voices or behaviour
Data and information
Processing and operating environment
The model processes the inputs. Whether it runs internally or with a service provider must be established for the specific implementation.
Storage and further use
Operational measurements such as delivery times and acceptance rates influence future allocations and ratings.
Results and data sharing
The system assigns jobs and shifts itself and can set bonuses or suspend accounts.
Using a provider does not automatically determine where processing takes place. Storage location, subcontractors, deletion periods and use for training must be checked for the specific deployment.
Assess risks and determine measures
From understanding the system to analysing risks
- Define the system boundary
Record data, participants, interfaces and permissions. Resolve uncertain characteristics in the model.
- Investigate threats
Relate review candidates to actual data flows: what could happen, and who would be affected?
- Assess risk
Assess impacts and likelihood in the deployment context. The number of review candidates is not a risk score.
- Provide evidence of measures
Select appropriate controls, assign responsibilities and define effectiveness checks. Document residual risks and approvals.
Key threats in the application context
- 4 particularly important
- 4 typical
- 8 depending on the system
- 8 additional threats for every AI system
A total of 24 threats from the threat catalogue may be relevant to this use case.
Also review: whether the system disadvantages groups
Check whether selection, ranking or assessment disadvantages people with protected characteristics, such as sex, age or origin.
Priority review points
If the system prioritises short delivery times, it allocates jobs so tightly that little time remains for breaks. Delivery times fall, while drivers' health and safety suffer.
The system can withdraw shifts or suspend accounts when measurements fall below a threshold. A measurement error, such as an incorrect location, can then cost someone their income without a review.
Supervisors use performance figures in discussions and appraisals without knowing how they were calculated. Traffic jams or long waits at restaurants then go unaccounted for.
Operational measurements guide future allocations. Falsifying location data or deliberately rejecting jobs changes the values the system uses to assign work to everyone else.
4 further review candidates in the typical design
Other relevant threats
Targeted Poisoning / Label Poisoning
Falsified training labels can deliberately shift model outputs.
Listed because of this characteristic
- Uses operational feedback
Targeted queries can recover some personal or protected training data from the model.
Listed because of these characteristics
- Processes personal data
- Processes confidential data
Sensitive Information Disclosure
Confidential content can appear in answers, supplied context or logs.
Listed because of these characteristics
- Processes personal data
- Processes confidential data
Exfiltration from ML Application
Protected data can leak through tools, network connections, storage or insiders.
Listed because of this characteristic
- Processes confidential data
Up to 8 further threats may apply, depending on your system design. Nine threats need to be reviewed for every AI system. One of these is particularly important here and is therefore listed above.
System features and open questions
Each Yes adds further threats to your checklist, up to 8 in total. If you cannot yet answer a question, include its threats in your review.
- Added by this Yes
- Also requires a Yes to another question
How your system is built
- Learns from your own data1 threat, plus 1 only in combination with other characteristics
- Uses another provider's model1 threat
How you operate the system
- Accepts queries directly through an interface4 threats
- Accessible over a network2 threats, plus 1 only in combination with other characteristics
- Incurs costs with each query1 threat
- Shares computing resources with other customers or applications1 threat
- Used alongside unapproved AI services1 threat
How your system is built
Does your organisation train or fine-tune the model itself using its own data?
If yes, add
Does the model or its base model come from another provider, even if you host it?
If yes, add
How you operate the system
These questions arise for almost every AI system.
Can people or programs query the model directly, for example through an interface?
If yes, add
Is the system accessible over a network or dependent on network services?
In delivery services, workers receive assignments through a phone app.
If yes, add
Do you pay per query or unit of computing time, so costs increase with use?
If yes, add
Does the system share servers or computing resources with other customers or applications?
If yes, add
Do staff also use AI services for this task that your organisation has not approved?
If yes, add
Controls and implementation
Measures for the priority review areas are expanded. Each linked measure explains how to implement it.
Review priorityMisalignment3 measures
Review priorityExcessive Agency3 measures
Review priorityOverreliance3 measures
Review priorityTraining Data Poisoning2 measures
Targeted Poisoning / Label Poisoning3 measures
Privacy Attacks3 measures
Sensitive Information Disclosure3 measures
Exfiltration from ML Application3 measures
Controls for the remaining threats are available on their pages in the threat catalogue.
AI Act provisions
-
Art. 4: AI literacy
For every AI system Applies since 2 February 2025.
-
Art. 5 para. 1 point e, f, g, h: Prohibited biometric practices
Depending on use Review trigger since 2 February 2025.
-
Annex III no. 4 point b: Employment decisions
In typical use Review trigger from 2 December 2027.
Applies to every AI system
Art. 4: AI literacy
Applies if staff or appointed representatives operate or use AI systems.
Applies since 2 February 2025.
Obligations and details for Art. 4: AI literacy
- Who bears the obligations
- Provider, Deployer, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
May apply in typical use
Annex III no. 4 point b: Employment decisions
Applies if the system allocates tasks based on behaviour or personal characteristics, monitors workers' performance and behaviour, or influences decisions on working conditions, promotion or dismissal.
Applies from 2 December 2027.
Obligations and details for Annex III no. 4 point b: Employment decisions
- The general condition is
- Applies if the system influences working conditions, promotion or dismissal, allocates tasks by behaviour or personal characteristics, or monitors workers' performance and behaviour.
- Who bears the obligations
- Provider, Deployer, Product manufacturer, Importer, Distributor, Authorised representative
Detailed obligations
- Document the classification with reasons
- Check whether you become a provider
- Establish and operate a risk management system
- Register the exemption under Art. 6(3)
- Use the system in line with the instructions
- Assign human oversight to suitable people
- Inform workers and their representatives in advance
- Inform affected persons about the use
- Explain decisions on request
May apply in particular uses
Art. 5 para. 1 point e, f, g, h: Prohibited biometric practices
Applies if the system infers workers' emotions from their faces or voices, for example to detect mood or anger in a call centre.
Applies since 2 February 2025.
Obligations and details for Art. 5 para. 1 point e, f, g, h: Prohibited biometric practices
- The general condition is
- Applies if the system indiscriminately collects facial images from the internet or video, recognises workplace or educational emotions, categorises people biometrically by sensitive attributes, or identifies people in public spaces in real time for law enforcement.
- Exception
- Workplace and educational emotion recognition is permitted for medical or safety reasons. Real-time law enforcement identification has narrowly defined exceptions.
- Who bears the obligations
- Provider, Deployer, Importer, Distributor, Authorised representative, Product manufacturer, Provider of a general-purpose AI model
Detailed obligations
Other legal requirements
Directive (EU) 2024/2831
The Directive limits automated monitoring and decision-making in platform work and requires human review of important decisions. Member States must transpose it by 2 December 2026.
Classify your system with the quick checker All obligations in the obligations compass
Cross-cutting threats
Review these threats for every AI system, whatever its purpose.
Atypical system features
View 17 further questions
If any of these features does apply to your system, add the listed threats.
Does the system generate its own text, such as answers, drafts or summaries?
The system produces allocations, measurements and ratings, rather than freely generated text.
If yes, add
Does the system generate or edit images, sounds, voices or videos?
If yes, add
Does the system create new content from open-ended requests that people read, see or hear directly? Translation, reading aloud and transcription do not count.
If yes, add
Can users give the system natural language tasks that it then performs?
The system works with jobs, shift schedules and measurements, rather than natural language instructions.
If yes, add
Does the system search its own collection of documents or records for relevant content?
If yes, add
Does the system formulate answers from retrieved documents?
If yes, add
Does the system use a knowledge graph storing concepts and their relationships?
If yes, add
Does the system remember content from earlier conversations or tasks and reuse it later?
If yes, add
Does the system call other programs, interfaces or search services itself, even if only to read?
If yes, add
Does the system determine its next steps itself based on earlier results?
If yes, add
Does the system delegate tasks to other AI agents or incorporate their results?
If yes, add
Is the system connected to tools or data sources through the Model Context Protocol (MCP)?
If yes, add
Do documents from several customers, departments or permission groups share one search store?
If yes, add
Can people outside your organisation use the system, including without login?
Only workers and their supervisors use the system, rather than the public.
If yes, add
Does the system process several input types together, such as text with images or speech?
If yes, add
Does the system generate code, scripts or database queries for further use?
If yes, add
Does the system execute programs, scripts or commands itself?
If yes, add
Sources
- Legislation, norm or standard
EU AI Act, Annex III, point 4(b)
The Act expressly lists AI systems that influence promotion or dismissal, allocate tasks, or monitor performance and behaviour in Annex III.
- Specialist study
OECD.AI, Algorithmic management in the workplace, 2025
A survey of over 6,000 companies, including companies in Germany, shows that algorithmic management tools are already widespread.
- Specialist study
IAB-Forum, Algorithmic management in app-based delivery services, 23 September 2025
The IAB surveyed workers at eleven delivery services in Germany.
- Other source
AlgorithmWatch, Join the Ride Side, 19 March 2024
AlgorithmWatch describes the use of algorithmic management at Lieferando.
Related use cases
Also in the field of Employment and education
- AI-assisted candidate screeningThe system uses CVs and cover letters to identify the best match for a role and suggest whom the company should invite for interview.
- Online proctoringThe system monitors online exam candidates through camera, microphone and screen and flags possible cheating.
- Automated assessment of exam performance (essay scoring)A model assesses written or spoken exam answers and awards marks, alone or alongside human examiners.
Often used together with
- Autonomous mobile robots in intralogisticsRobots independently move goods around a warehouse or accompany staff picking orders; a controller allocates orders and plans routes.
- Sales and demand forecastingRetailers use a model to predict how much of each item each store will sell, then turn those forecasts into suggested orders or completed orders.
Updated: 30 September 2026. Review checklist rules version 0.3.2, threat catalogue version 2026.10.24.2. How these pages are structured