AI threat catalogueHarmful ContentProduction
Sexual Content
The AI system produces sexually explicit or suggestive content in a context where it is inappropriate, unwanted, or unlawful. The possible depiction of minors is especially critical.
Description
The model returns sexual content that does not belong in its deployment context, whether on request or through a bypass of its safety controls. Multimodal systems can produce such content as images as well. The gravest cases are material classifiable as child sexual abuse material (CSAM) and intimate images of real people created without their consent. Abuse depictions are a criminal offense even when they are purely synthetic, that is, entirely invented by the model.
Possible impact
Producing abuse material is a criminal offense in Germany and the EU, including AI-generated material, and creates very high liability and mandatory-reporting risk for the operator. It also brings platform bans, reputational damage, and directly concerns the protection of children and other affected people.
Example
A publicly accessible image generator is steered into producing suggestive depictions of a person who appears to be a minor, or a chatbot writes explicit text inside an application intended for young people.
Recommended mitigations (5)
Every mitigation states its control type, effect, implementation level and the reason for the classification.
Strict content filtering for sexual contentTechnical
- Effect
- Preventive
- Implementation level
- Application, API & agents
- Reason for the classification
- “Strict content filtering for sexual content” is primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.
NSFW detection modelsTechnical
- Effect
- Detective
- Implementation level
- Model & training, Application, API & agents
- Reason for the classification
- “NSFW detection models” is primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.
Age verification (where relevant)Technical
- Effect
- Preventive, Detective
- Implementation level
- Application, API & agents, Organization
- Complementary control type
- Governance & compliance
- Reason for the classification
- “Age verification (where relevant)” is primarily technical: Machine-enforced identity, permission, or scope rules constrain unauthorized access and actions; complemented by rules and oversight.
Explicit policy enforcementTechnical
- Effect
- Preventive
- Implementation level
- Application, API & agents, Organization
- Complementary control type
- Governance & compliance
- Reason for the classification
- “Explicit policy enforcement” is primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing; complemented by rules and oversight.
CSAM detection and reportingTechnical
- Effect
- Detective
- Implementation level
- Application, API & agents, Organization, Use & operations
- Complementary control type
- Governance & compliance
- Reason for the classification
- “CSAM detection and reporting” is primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators; complemented by rules and oversight.
Framework mappings
Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.
Verified references (5)
Every reference states the framework, the exact location and the publishing organisation.
- OWASP LLM Top 10 LLM01:2025 Prompt InjectionLLM01:2025 Prompt Injection, official category page OWASP FoundationOriginal
- NIST AI RMF Section 2.11 Obscene, Degrading, and/or Abusive ContentSection 2.11, pp. 11–12 National Institute of Standards and Technology (NIST)Original
- MITRE ATLAS AML.T0048.002 Societal HarmATLAS.yaml technique object with id AML.T0048.002 (pinned release v5.6.0) MITREOriginal
- EU AI Act Article 55(1)(b) Obligations of providers of general-purpose AI models with systemic riskArticle 55(1)(b) European Union (EUR-Lex)Original
- BSI R5 Problematische und verzerrte Ausgaben (Text, Bild, Video)Kap. 4, R5, p. 16 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
Related threats
More entries from the topic group Harmful Content.
Assess this threat in your own system
The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.
Cite this entry
For reports, policies or internal documents; the link leads directly to this entry.
“Sexual Content”. Versatile AI Risk Assessment, AI threat catalogue, as of July 2026. https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/sexual-content/