AI threat catalogueMalicious Use for Attacks, Fraud and DisinformationProduction
Social Engineering
Attackers use AI to produce deceptively authentic, personally tailored scam messages, calls and pretext stories at scale. Familiar warning signs such as clumsy language disappear, making the deception considerably more convincing.
Description
Social engineering tricks people into revealing confidential information, making payments or installing malware. Generative AI amplifies this tactic considerably: language models write flawless phishing messages (fake communications designed to prompt a harmful action) tailored to individual recipients and their company, and provide scripts for fraudulent phone calls. Voice and video generators additionally imitate real people, such as managers or business partners. The attacks target people rather than technology, arriving by email, phone, messenger or video call. AI lowers the entry barrier and increases the volume, speed and quality of such attacks.
Possible impact
A successful deception can lead to fraudulent payments, stolen credentials and, in turn, compromised systems and data leaks. Beyond the financial damage, reporting and liability questions arise, for example when personal data is exposed. Staff in finance, HR and support roles are particularly at risk, and the trust of customers and partners in the company’s communication suffers as well.
Example
The accounting team receives an email that precisely matches the tone and writing style of the CEO; shortly afterwards a call arrives using a cloned version of the CEO’s voice: a supposedly confidential acquisition requires an immediate transfer. Attacks of this kind, known as CEO fraud, become far more convincing with AI-generated text and voices.
Recommended mitigations (5)
Every mitigation states its control type, effect, implementation level and the reason for the classification.
Detection of impersonation/pretext patternsTechnical
- Effect
- Detective
- Implementation level
- Application, API & agents
- Reason for the classification
- “Detection of impersonation/pretext patterns” is primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.
Refusal for deceptive content generationTechnical
- Effect
- Preventive
- Implementation level
- Application, API & agents
- Reason for the classification
- “Refusal for deceptive content generation” is primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.
Identity verification in sensitive workflowsTechnical
- Effect
- Preventive, Detective
- Implementation level
- Application, API & agents, Use & operations
- Complementary control type
- Organizational & process-based
- Reason for the classification
- “Identity verification in sensitive workflows” is primarily technical: Machine-enforced identity, permission, or scope rules constrain unauthorized access and actions; complemented by binding workflows.
User awareness trainingPeople & competence
- Effect
- Preventive
- Implementation level
- Organization, Use & operations
- Complementary control type
- Organizational & process-based
- Reason for the classification
- “User awareness training” is primarily a people-and-competence control: Trained users recognize deception and abuse patterns; audience, repetition, and effectiveness requirements support application.
Anti-phishing detectionTechnical
- Effect
- Detective
- Implementation level
- Application, API & agents
- Reason for the classification
- “Anti-phishing detection” is primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.
Framework mappings
Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.
Verified references (7)
Every reference states the framework, the exact location and the publishing organisation.
- NIST AI RMF Section 2.8 Information IntegritySection 2.8, pp. 9–10 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF Section 2.9 Information SecuritySection 2.9, pp. 10–11 National Institute of Standards and Technology (NIST)Original
- MITRE ATLAS AML.T0048.002 Societal HarmATLAS.yaml technique object with id AML.T0048.002 (pinned release v5.6.0) MITREOriginal
- EU AI Act Article 5(1)(a) Prohibited AI practicesArticle 5(1)(a); where the catalogue cites exploitation, compare Article 5(1)(b) European Union (EUR-Lex)Original
- EU AI Act Article 55(1)(b) Obligations of providers of general-purpose AI models with systemic riskArticle 55(1)(b) European Union (EUR-Lex)Original
- EU AI Act Article 9(1), 9(2)(a), 9(2)(d) Risk management systemArticle 9(1), 9(2)(a), 9(2)(d), read with Article 9(3) European Union (EUR-Lex)Original
- BSI R11 Vortäuschen einer (medialen) Identität (Text, Bild, Video)Kap. 4, R11, p. 19 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
Related threats
More entries from the topic group Malicious Use for Attacks, Fraud and Disinformation.
Assess this threat in your own system
The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.
Cite this entry
For reports, policies or internal documents; the link leads directly to this entry.
“Social Engineering”. Versatile AI Risk Assessment, AI threat catalogue, as of July 2026. https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/social-engineering/