AI threat catalogueMalicious Use for Attacks, Fraud and DisinformationProduction
Malicious Software
Attackers use AI models to generate or improve working malware and attack tooling. This lowers the entry barrier: even perpetrators without advanced programming skills can prepare attacks this way.
Description
AI models with coding capabilities can write not only useful programs but also malicious code: for example ransomware, spyware, or exploit code, meaning code that deliberately takes advantage of security vulnerabilities. Attackers bypass the models’ built-in safeguards through jailbreaks (inputs that override a model’s safety measures) or switch to models without such restrictions. AI also helps to find vulnerabilities in software quickly and partly automatically and to turn them into usable attack paths. So far, security authorities have mainly observed an acceleration and simplification of existing attack methods; even this, however, noticeably lowers the entry barrier for perpetrators.
Possible impact
Companies must expect more attacks, developed faster, because the pool of potential perpetrators grows and attack tooling becomes easier to obtain. If such malware reaches the organization, the consequences include business interruption, encrypted or stolen data, and high recovery costs. If a company’s own AI system is misused to generate malicious code, the operator additionally faces liability and reputational questions.
Example
An attacker without advanced programming skills has a language model build a working piece of malware, including mechanisms to disguise it, and sends it to the HR department as a rigged job application attachment.
Recommended mitigations (5)
Every mitigation states its control type, effect, implementation level and the reason for the classification.
Refusal training for malware codeTechnical
- Effect
- Preventive
- Implementation level
- Model & training
- Reason for the classification
- “Refusal training for malware code” is primarily technical: A model, training, or data-processing method directly changes system behavior or robustness.
Static analysis of generated codeTechnical
- Effect
- Detective
- Implementation level
- Application, API & agents, Use & operations
- Reason for the classification
- “Static analysis of generated code” is primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.
Dual-use evaluationGovernance & compliance
- Effect
- Detective
- Implementation level
- Model & training, Organization, Use & operations
- Complementary control type
- Organizational & process-based
- Reason for the classification
- “Dual-use evaluation” is primarily a governance and compliance control: Binding rules, control objectives, or oversight define permitted use and accountability; complemented by binding workflows.
Monitoring for malicious code patternsTechnical
- Effect
- Detective
- Implementation level
- Application, API & agents, Use & operations
- Complementary control type
- Organizational & process-based
- Reason for the classification
- “Monitoring for malicious code patterns” is primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators; complemented by binding workflows.
Red-teaming with security researchersOrganizational & process-based
- Effect
- Detective
- Implementation level
- Model & training, Application, API & agents, Use & operations
- Complementary control type
- People & competence, Technical
- Reason for the classification
- “Red-teaming with security researchers” is primarily organizational and process-based: A planned, repeatable assessment with ownership and documented follow-up creates the protective effect; complemented by human expertise and judgment as well as technical implementation.
Framework mappings
Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.
Verified references (6)
Every reference states the framework, the exact location and the publishing organisation.
- OWASP LLM Top 10 LLM01:2025 Prompt InjectionLLM01:2025 Prompt Injection, official category page OWASP FoundationOriginal
- NIST AI RMF Section 2.9 Information SecuritySection 2.9, pp. 10–11 National Institute of Standards and Technology (NIST)Original
- EU AI Act Article 55(1)(b) Obligations of providers of general-purpose AI models with systemic riskArticle 55(1)(b) European Union (EUR-Lex)Original
- EU AI Act Article 9(1), 9(2)(a), 9(2)(d) Risk management systemArticle 9(1), 9(2)(a), 9(2)(d), read with Article 9(3) European Union (EUR-Lex)Original
- BSI R12 Wissenssammlung und -aufbereitung im Kontext krimineller Aktivitäten (Text, Bild)Kap. 4, R12, p. 20 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BSI R14 Generierung und Verbesserung von Malware (Text)Kap. 4, R14, p. 22 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
Related threats
More entries from the topic group Malicious Use for Attacks, Fraud and Disinformation.
Assess this threat in your own system
The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.
Cite this entry
For reports, policies or internal documents; the link leads directly to this entry.
“Malicious Software”. Versatile AI Risk Assessment, AI threat catalogue, as of July 2026. https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/malicious-software/