Versatile AI Risk Assessment

AI threat catalogue Harmful Content Production

Sexual Content

The AI system produces sexually explicit or suggestive content in a context where it is inappropriate, unwanted, or unlawful. The possible depiction of minors is especially critical.

As of: October 2026 · Catalogue version 2026.10.24.2 · 5 mitigations · 7 verified sources

Description

The model returns sexual content that does not belong in its deployment context, whether on request or through a bypass of its safety controls. Multimodal systems can produce such content as images as well. The gravest cases are material classifiable as child sexual abuse material (CSAM) and intimate images of real people created without their consent. Abuse depictions are a criminal offense even when they are purely synthetic, that is, entirely invented by the model.

Possible impact

Producing abuse material is a criminal offense in Germany and the EU, including AI-generated material, and creates very high liability and mandatory-reporting risk for the operator. It also brings platform bans, reputational damage, and directly concerns the protection of children and other affected people.

Example

A publicly accessible image generator is steered into producing suggestive depictions of a person who appears to be a minor, or a chatbot writes explicit text inside an application intended for young people.

Recommended mitigations (5)

Compare effects and implementation levels. Expand each entry for the additional classification and rationale.

  • Strict content filtering for sexual content

    Technical
    Effect
    Preventive
    Implementation level
    Application, API & agents
    Classification and rationale

    Primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.

  • NSFW detection models

    Technical
    Effect
    Detective
    Implementation level
    Model & training, Application, API & agents
    Classification and rationale

    Primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.

  • Age verification (where relevant)

    Technical
    Effect
    Preventive, Detective
    Implementation level
    Application, API & agents, Organization
    Classification and rationale

    Complementary control typeGovernance & compliance

    Primarily technical: Machine-enforced identity, permission, or scope rules constrain unauthorized access and actions; complemented by rules and oversight.

  • Explicit policy enforcement

    Technical
    Effect
    Preventive
    Implementation level
    Application, API & agents, Organization
    Classification and rationale

    Complementary control typeGovernance & compliance

    Primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing; complemented by rules and oversight.

  • CSAM detection and reporting

    Technical
    Effect
    Detective
    Implementation level
    Application, API & agents, Organization, Use & operations
    Classification and rationale

    Complementary control typeGovernance & compliance

    Primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators; complemented by rules and oversight.

Framework mappings

Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.

OWASP LLM Top 10 LLM01:2026NIST AI RMF Section 2.11MITRE ATLAS AML.T0048.002EU AI Act Article 5(1)(ba) · Article 5(1)(bb) · Article 55(1)(b)BSI R5

Verified references (7)

Every reference states the framework, the exact location and the publishing organisation.

EU AI Act obligations related to this threat

The obligations from the obligations compass whose articles the catalogue maps to this threat. Each leads to its entry with core, action step and evidence; whether it applies to you depends on role and risk class.

  • Refrain from the prohibited practices Art. 5 Abs. 1 · applies from 2 February 2025

    Statutory roles: Provider, Deployer, Importer, Distributor, Authorised representative, Product manufacturer, Provider of a general-purpose AI model. Classes: Prohibited practice.

    These prohibitions have applied since 2 February 2025 and are unaffected by the deferral of the high-risk obligations. Infringements carry the highest fine bracket of the regulation.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • New prohibition: intimate depictions without consent Art. 5 Abs. 1 UAbs. 1 Buchst. ba, Art. 5 Abs. 1a, 1b · applies from 2 December 2026 · amended by Reg. 2026/1744

    Statutory roles: Provider, Deployer, Importer, Distributor, Authorised representative, Product manufacturer, Provider of a general-purpose AI model. Classes: Prohibited practice, System subject to transparency obligations.

    This prohibition is new and was inserted by Regulation (EU) 2026/1744. It applies from 2 December 2026. For deployers, the prohibition applies only where the system is used for that very purpose.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • New prohibition: child sexual abuse material Art. 5 Abs. 1 UAbs. 1 Buchst. bb · applies from 2 December 2026 · amended by Reg. 2026/1744

    Statutory roles: Provider, Deployer, Importer, Distributor, Authorised representative, Product manufacturer, Provider of a general-purpose AI model. Classes: Prohibited practice.

    Also newly inserted by Regulation (EU) 2026/1744, applying from 2 December 2026.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Evaluate and mitigate systemic risks Art. 55 · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

All 62 obligations by role, risk class and deadline in the obligations compass

More entries from the topic group Harmful Content.

Hate Speech and Discrimination The AI system produces content that demeans individuals or groups on the basis of protected characteristics such as origin, gender, … Profanity The AI system produces vulgar or offensive language in a setting where this is unprofessional or against policy, such as customer support, … Violence / Unsafe Actions The AI system depicts violence, glorifies it, or provides instructions for dangerous acts, such as building weapons or carrying out risky …

Matching use cases

This list shows where to review this threat, rather than how high the risk is.

Particularly important for:

  • Image, video and voice generationSomeone uploads a colleague's photograph and requests a revealing image. Without safeguards, this produces a non-consensual intimate depiction that seriously harms the person.

For 7 further use cases, relevance depends on the system design, for example whether it generates new content from open-ended requests or is also available to people outside your organisation. Show these use cases

Frequently asked questions

What is Sexual Content?
The AI system produces sexually explicit or suggestive content in a context where it is inappropriate, unwanted, or unlawful. The possible depiction of minors is especially critical.
Which mitigations help against Sexual Content?
The catalogue lists 5 mitigations, each with control type, effect and implementation level. First among them are Strict content filtering for sexual content, NSFW detection models and Age verification (where relevant).
Which EU AI Act obligations does Sexual Content touch?
The catalogue maps this threat to Art. 5 Abs. 1, Art. 5 Abs. 1 UAbs. 1 Buchst. ba, Art. 5 Abs. 1 UAbs. 1 Buchst. bb and Art. 55 of the EU AI Act. In the obligations compass this leads to: Refrain from the prohibited practices, New prohibition: intimate depictions without consent, New prohibition: child sexual abuse material and Evaluate and mitigate systemic risks. Whether an obligation applies depends on role, risk class and deadline.

Assess this threat in your own system

The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.

Start the live demo See the free modules

Cite this entry

For reports, policies or internal documents; the link leads directly to this entry.

“Sexual Content”. Versatile AI Risk Assessment, AI threat catalogue, as of October 2026.
https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/sexual-content/

← Back to the full catalogue