AI threat catalogueReliability and Responsible UseProduction
Overreliance
People tend to accept convincing-sounding AI outputs without checking them (automation bias). When employees or downstream systems rely uncritically on AI for important decisions, errors go undetected and cause real-world harm.
Description
AI systems phrase their answers fluently and confidently even when the content is wrong. This is exactly what fuels automation bias: the human tendency to trust the results of an automated system more than one's own judgement. This threat does not come from attackers; it arises in everyday work, for example under time pressure or when knowledge about the limits of AI is missing. It becomes especially critical when AI outputs feed into follow-up decisions or other systems without human review: a single undetected error then propagates and compounds. A decline in model quality also stays invisible for as long as nobody questions the results.
Possible impact
Poor decisions based on unchecked AI outputs first hit the people affected, such as applicants, customers, or patients, and then fall back on the company. Financial losses, liability questions, and reputational damage follow. The EU AI Act requires effective human oversight for high-risk systems, and the GDPR places strict limits on fully automated individual decisions.
Example
An HR department has incoming applications pre-ranked by an AI system and adopts the ranking as it is. Only months later does it emerge that the system systematically screened out suitable candidates and that nobody had ever spot-checked its recommendations.
Recommended mitigations (5)
Every mitigation states its control type, effect, implementation level and the reason for the classification.
Clear confidence indicators in UITechnical
- Effect
- Preventive, Detective
- Implementation level
- Application, API & agents, Use & operations
- Reason for the classification
- “Clear confidence indicators in UI” is primarily technical: The application makes uncertainty, system boundaries, or safe next steps visible and supports informed decisions.
User training on AI limitationsPeople & competence
- Effect
- Preventive
- Implementation level
- Organization, Use & operations
- Complementary control type
- Organizational & process-based
- Reason for the classification
- “User training on AI limitations” is primarily a people-and-competence control: Human knowledge, attention, or professional judgment produces the protective decision; complemented by binding workflows.
Human verification for high-stakes decisionsOrganizational & process-based
- Effect
- Preventive, Detective
- Implementation level
- Organization, Use & operations
- Complementary control type
- People & competence
- Reason for the classification
- “Human verification for high-stakes decisions” is primarily organizational and process-based: A binding workflow requires an accountable human decision before use or execution; complemented by human expertise and judgment.
Output provenance and source attributionTechnical
- Effect
- Preventive, Detective
- Implementation level
- Application, API & agents, Use & operations
- Complementary control type
- Organizational & process-based
- Reason for the classification
- “Output provenance and source attribution” is primarily technical: Cryptographic or machine-verifiable properties protect confidentiality, integrity, or provenance; complemented by binding workflows.
Explainability featuresTechnical
- Effect
- Detective
- Implementation level
- Model & training, Application, API & agents, Use & operations
- Reason for the classification
- “Explainability features” is primarily technical: The application makes uncertainty, system boundaries, or safe next steps visible and supports informed decisions.
Framework mappings
Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.
Verified references (11)
Every reference states the framework, the exact location and the publishing organisation.
- OWASP LLM Top 10 LLM09:2025 MisinformationLLM09:2025 Misinformation, official category page OWASP FoundationOriginal
- NIST AI RMF Section 2.2 ConfabulationSection 2.2, p. 6 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF Section 2.7 Human-AI ConfigurationSection 2.7, p. 9 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF GOVERN 3.2 GOVERN 3.2GOVERN 3.2, p. 23 National Institute of Standards and Technology (NIST)Original
- EU AI Act Article 13(1), 13(3)(b)(ii), (iv), (v) Transparency and provision of information to deployersArticle 13(1), 13(3)(b)(ii), (iv), (v) European Union (EUR-Lex)Original
- EU AI Act Article 14(4)(b) Human oversightArticle 14(4)(b) European Union (EUR-Lex)Original
- EU AI Act Article 4 AI literacyArticle 4 European Union (EUR-Lex)Original
- BSI R8 Automation Bias (Text, Bild, Video)Kap. 4, R8, p. 17 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BIML BIML-LLM LLMtop10:9 Model TrustworthinessPDF p. 13, [LLMtop10:9:model trustworthiness] Berryville Institute of Machine Learning (BIML)Original
- BIML BIML-LLM output:12 OverconfidencePDF p. 20, [output:12:overconfidence] Berryville Institute of Machine Learning (BIML)Original
- BIML BIML78 system:2 OverconfidencePDF p. 25, [system:2:overconfidence] Berryville Institute of Machine Learning (BIML)Original
Related threats
More entries from the topic group Reliability and Responsible Use.
Assess this threat in your own system
The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.
Cite this entry
For reports, policies or internal documents; the link leads directly to this entry.
“Overreliance”. Versatile AI Risk Assessment, AI threat catalogue, as of July 2026. https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/overreliance/