Overreliance

Belongs on the checklist for every AI system. What to review for every AI system

Particularly important for:

• Generative AI assistant for staffDrafts sound confident and read smoothly. People who accept them under time pressure may stop checking content and sources.
• Speech recognition and transcriptionThe transcript reads smoothly and looks complete. Approval without checking the recording misses misheard figures, names and negatives.
• AI meeting minutes and conversation documentationThe draft is ready immediately and looks complete. Under time pressure, staff sign it off without checking it against their recollection.
• Intelligent document processing (IDP)Approving many extracted records every day can become a cursory task. Wrong amounts or allocations then go unnoticed.
• Automated anonymisation and redactionMost suggestions are correct, so approval becomes routine. Missed passages are then discovered only after publication.
• E-discovery and technology-assisted review (TAR)Review ends when the model expects few relevant documents to remain. If that estimate is wrong, important records remain unread and the gap goes unnoticed.
• Text analytics for feedback and consultation responsesThe theme overview looks complete and is ordered by frequency. Rare but serious indications, such as danger, may never be read in full.
• Customer service chatbots and voicebotsCustomers treat the bot's information as binding because it speaks for the company. Staff accept its summary of a request without reading the conversation.
• Credit scoring and creditworthiness assessmentA caseworker may reject a score just below the threshold without considering the person's situation. An estimate becomes a decision, even if the score is wrong.
• Money laundering detection and sanctions screeningWith many false alarms, specialists begin dismissing alerts hastily and treating unflagged accounts as normal. Incorrect model assessments then go unnoticed.
• Life and health insurance risk assessment and pricingUnderwriters may accept an unfavourable risk class without reviewing health information. The person then receives a surcharge or rejection based only on an estimate.
• AI-assisted insurance claims handlingStaff approving many settlement proposals daily may start waving them through. Inconsistencies missed by the system then escape human review too.
• AI-assisted candidate screeningRecruiters often look only at the top-ranked candidates. Those placed lower have little chance, even when the ranking is wrong.
• Algorithmic managementSupervisors use performance figures in discussions and appraisals without knowing how they were calculated. Traffic jams or long waits at restaurants then go unaccounted for.
• Online proctoringInvigilators focus on flagged passages and accept the suspicion. Disability, a child in the room or poor lighting can then unfairly cast suspicion on candidates.
• Automated assessment of exam performance (essay scoring)A second examiner who sees the automated score first may be influenced by it. Human assessment then ceases to check it independently.
• Remote biometric identification (facial recognition)The system returns the closest match even if the person sought is absent from the database. Investigators who rely on it can place an uninvolved person under suspicion, as the Detroit incidents show.
• Emotion recognitionFacial expressions do not reliably reveal feelings and differ across cultures and individuals. Trusting the output may label a tense traveller suspicious or a calm worker unmotivated.
• Risk-based review selection for applications and benefitsReviewers can assume something is wrong with a selected case. Selection for review becomes suspicion, as in the Dutch childcare benefits scandal.
• Automated property valuation (AVM)With thousands of properties, valuers often scrutinise only unusual estimates. An unremarkable but incorrect value enters the notice, leaving the owner to challenge it.
• Police data analysis platformsA graph connection looks like a finding even if based only on a shared address or witness statement. Accepting it unchecked can draw an uninvolved person into proceedings.
• AI assistance in high-volume court proceedingsWith many similar claims, simply accepting extracted features and drafts is tempting. If the system misses a distinguishing detail, the judgment does not fit the case.
• Language and dialect recognition in asylum proceduresA dialect probability looks like a measurement but says nothing about where someone grew up. Treating it as proof can make someone whose speech reflects several regions seem untrustworthy.
• Automatic number plate recognition (ANPR)Enforcement staff see many matches each day and may begin confirming them without close inspection. A misread digit then produces a notice against an uninvolved person.
• Intelligent video analytics in public spacesControl centre staff focus on scenes the system flags. Undetected events go unwatched, and harmless rough play may be treated as an attack.
• AI-assisted medical image analysisClinicians reviewing many AI-marked images may increasingly focus on marked regions. Unmarked findings are then more easily missed.
• Clinical decision support (such as early sepsis detection)A team relying on sepsis alerts may quickly consider an unflagged patient stable. Treatment then begins too late.
• AI support for emergency call handlingA dispatcher relying on cardiac arrest alerts may stop asking targeted questions when none appears. Telephone resuscitation instructions then begin too late.
• Predictive maintenanceMaintenance teams relying on predictions may extend fixed inspection intervals. Damage no sensor detects becomes apparent only at failure, disrupting operations and potentially endangering people at points or energy facilities.
• Visual and acoustic quality inspectionRelying on the camera to find every defect can lead to dropping manual sampling. Defect types the model has never seen remain undetected.
• Automated driving and driver assistanceDrivers become accustomed to the system handling the situation and turn their attention elsewhere. If it requests a takeover or fails unnoticed, too little time remains to intervene.
• Coding assistantSuggestions that fit smoothly and compile successfully are easily waved through without thorough review. Vulnerabilities such as missing input validation then enter the code.
• AI-based cyberattack detectionWhen the system reports nothing, a network is soon assumed to be clean and people stop hunting for attackers themselves. Attacks the model does not recognise as deviations remain undetected.