Versatile AI Risk Assessment

AI threat catalogue Malicious Use for Attacks, Fraud and Disinformation Production

Malicious Software

Attackers use AI models to generate or improve working malware and attack tooling. This lowers the entry barrier: even perpetrators without advanced programming skills can prepare attacks this way.

As of: October 2026 · Catalogue version 2026.10.24.2 · 5 mitigations · 6 verified sources

Description

AI models with coding capabilities can write not only useful programs but also malicious code: for example ransomware, spyware, or exploit code, meaning code that deliberately takes advantage of security vulnerabilities. Attackers bypass the models’ built-in safeguards through jailbreaks (inputs that override a model’s safety measures) or switch to models without such restrictions. AI also helps to find vulnerabilities in software quickly and partly automatically and to turn them into usable attack paths. So far, security authorities have mainly observed an acceleration and simplification of existing attack methods; even this, however, noticeably lowers the entry barrier for perpetrators.

Possible impact

Companies must expect more attacks, developed faster, because the pool of potential perpetrators grows and attack tooling becomes easier to obtain. If such malware reaches the organization, the consequences include business interruption, encrypted or stolen data, and high recovery costs. If a company’s own AI system is misused to generate malicious code, the operator additionally faces liability and reputational questions.

Example

An attacker without advanced programming skills has a language model build a working piece of malware, including mechanisms to disguise it, and sends it to the HR department as a rigged job application attachment.

Recommended mitigations (5)

Compare effects and implementation levels. Expand each entry for the additional classification and rationale.

  • Refusal training for malware code

    Technical
    Effect
    Preventive
    Implementation level
    Model & training
    Classification and rationale

    Primarily technical: A model, training, or data-processing method directly changes system behavior or robustness.

  • Static analysis of generated code

    Technical
    Effect
    Detective
    Implementation level
    Application, API & agents, Use & operations
    Classification and rationale

    Primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.

  • Dual-use evaluation

    Governance & compliance
    Effect
    Detective
    Implementation level
    Model & training, Organization, Use & operations
    Classification and rationale

    Complementary control typeOrganizational & process-based

    Primarily a governance and compliance control: Binding rules, control objectives, or oversight define permitted use and accountability; complemented by binding workflows.

  • Monitoring for malicious code patterns

    Technical
    Effect
    Detective
    Implementation level
    Application, API & agents, Use & operations
    Classification and rationale

    Complementary control typeOrganizational & process-based

    Primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators; complemented by binding workflows.

  • Red-teaming with security researchers

    Organizational & process-based
    Effect
    Detective
    Implementation level
    Model & training, Application, API & agents, Use & operations
    Classification and rationale

    Complementary control typePeople & competence, Technical

    Primarily organizational and process-based: A planned, repeatable assessment with ownership and documented follow-up creates the protective effect; complemented by human expertise and judgment as well as technical implementation.

Framework mappings

Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.

OWASP LLM Top 10 LLM01:2026NIST AI RMF Section 2.9EU AI Act Article 55(1)(b) · Article 9(1), 9(2)(a), 9(2)(d)BSI R12 · R14

Verified references (6)

Every reference states the framework, the exact location and the publishing organisation.

EU AI Act obligations related to this threat

The obligations from the obligations compass whose articles the catalogue maps to this threat. Each leads to its entry with core, action step and evidence; whether it applies to you depends on role and risk class.

  • Establish and operate a risk management system Art. 9 · applies from Annex III: 2 December 2027 · Annex I: 2 August 2028

    Statutory roles: Provider, Product manufacturer. Classes: High-risk under Annex III, High-risk as a product or safety component.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Evaluate and mitigate systemic risks Art. 55 · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

All 62 obligations by role, risk class and deadline in the obligations compass

Terms on this page

Glossary terms that occur in this entry. Every link leads to the full explanation.

  • Jailbreak Input that makes a model bypass its own safeguards.

More entries from the topic group Malicious Use for Attacks, Fraud and Disinformation.

Social Engineering Attackers use AI to produce deceptively authentic, personally tailored scam messages, calls and pretext stories at scale. Fraud Criminals use AI to commit fraud at greater scale and with greater credibility: through fake reviews, invented identities, forged … Disinformation AI makes it possible to mass-produce false or misleading content of convincing quality: fabricated news articles, manipulated images and …

Matching use cases

This list shows where to review this threat, rather than how high the risk is.

Particularly important for:

  • Coding assistantThe assistant suggests a nonexistent library, and attackers have published a malicious package under that exact name. Accepting the suggestion introduces malicious code into the organisation's software.

Typical for:

For 3 further use cases, relevance depends on the system design, for example whether it generates code or queries. Show these use cases

Frequently asked questions

What is Malicious Software?
Attackers use AI models to generate or improve working malware and attack tooling. This lowers the entry barrier: even perpetrators without advanced programming skills can prepare attacks this way.
Which mitigations help against Malicious Software?
The catalogue lists 5 mitigations, each with control type, effect and implementation level. First among them are Refusal training for malware code, Static analysis of generated code and Dual-use evaluation.
Which EU AI Act obligations does Malicious Software touch?
The catalogue maps this threat to Art. 9 and Art. 55 of the EU AI Act. In the obligations compass this leads to: Establish and operate a risk management system and Evaluate and mitigate systemic risks. Whether an obligation applies depends on role, risk class and deadline.

Assess this threat in your own system

The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.

Start the live demo See the free modules

Cite this entry

For reports, policies or internal documents; the link leads directly to this entry.

“Malicious Software”. Versatile AI Risk Assessment, AI threat catalogue, as of October 2026.
https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/malicious-software/

← Back to the full catalogue