Versatile AI Risk Assessment

AI threat catalogue Harmful Content Production

Hate Speech and Discrimination

The AI system produces content that demeans individuals or groups on the basis of protected characteristics such as origin, gender, religion, or disability, or that calls for their exclusion.

As of: October 2026 · Catalogue version 2026.10.24.2 · 5 mitigations · 10 verified sources

Description

Such output arises in three ways: on direct request, through a jailbreak (the deliberate circumvention of the safety controls built into the model), or unintentionally, when the model reproduces prejudice and bias absorbed from its training data. The range runs from stereotyping phrasing and disparaging language to incitement of hatred or violence against an identity group. Any channel in which the system generates free-form text is affected, including chatbots, assistants, and automated decisions.

Possible impact

Operators face reputational damage, legal exposure under anti-discrimination law such as the German General Equal Treatment Act (AGG), and regulatory consequences. Discriminatory output violates the fundamental right to non-discrimination and directly harms the people concerned. In automated processes such as recruitment, disadvantaging results can systematically exclude entire groups of people.

Example

A recruitment chatbot phrases a rejection in a way that demeans female applicants because of their gender, or a customer-service assistant answers a harmless question with a stereotyping statement about an ethnic group.

Recommended mitigations (5)

Compare effects and implementation levels. Expand each entry for the additional classification and rationale.

  • Content moderation filters

    Technical
    Effect
    Preventive
    Implementation level
    Application, API & agents
    Classification and rationale

    Primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.

  • Bias testing and evaluation

    Technical
    Effect
    Detective
    Implementation level
    Model & training, Use & operations
    Classification and rationale

    Primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.

  • Safety fine-tuning (RLHF)

    Technical
    Effect
    Preventive
    Implementation level
    Model & training
    Classification and rationale

    Primarily technical: A model, training, or data-processing method directly changes system behavior or robustness.

  • User feedback mechanisms

    Organizational & process-based
    Effect
    Detective
    Implementation level
    Application, API & agents, Use & operations
    Classification and rationale

    Complementary control typeTechnical, People & competence

    Primarily organizational and process-based: A defined reporting, triage, and handling workflow turns observations into traceable follow-up actions; complemented by technical implementation as well as human expertise and judgment.

  • Diverse training data

    Technical
    Effect
    Preventive
    Implementation level
    Data, Model & training
    Classification and rationale

    Primarily technical: A model, training, or data-processing method directly changes system behavior or robustness.

Framework mappings

Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.

OWASP LLM Top 10 LLM01:2026NIST AI RMF Section 2.3 · Section 2.6 · MEASURE 2.11MITRE ATLAS AML.T0048.002EU AI Act Article 10(2)(f), (g) · Article 55(1)(b)BSI R5BIML BIML-LLM output:11 · BIML78 system:1

Verified references (10)

Every reference states the framework, the exact location and the publishing organisation.

EU AI Act obligations related to this threat

The obligations from the obligations compass whose articles the catalogue maps to this threat. Each leads to its entry with core, action step and evidence; whether it applies to you depends on role and risk class.

  • Ensure data and data governance Art. 10, Art. 4a · applies from Annex III: 2 December 2027 · Annex I: 2 August 2028 · amended by Reg. 2026/1744

    Statutory roles: Provider, Product manufacturer. Classes: High-risk under Annex III, High-risk as a product or safety component.

    Regulation (EU) 2026/1744 introduced Art. 4a as an express legal basis for processing special categories of data to detect and correct bias, subject to six conditions such as pseudonymisation, strict access control, deletion after correction and no transfer to third parties. Art. 10(5) was deleted at the same time.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Evaluate and mitigate systemic risks Art. 55 · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

All 62 obligations by role, risk class and deadline in the obligations compass

Terms on this page

Glossary terms that occur in this entry. Every link leads to the full explanation.

  • Jailbreak Input that makes a model bypass its own safeguards.
  • Fine-tuning An existing model is trained further on your own data.

More entries from the topic group Harmful Content.

Profanity The AI system produces vulgar or offensive language in a setting where this is unprofessional or against policy, such as customer support, … Sexual Content The AI system produces sexually explicit or suggestive content in a context where it is inappropriate, unwanted, or unlawful. Violence / Unsafe Actions The AI system depicts violence, glorifies it, or provides instructions for dangerous acts, such as building weapons or carrying out risky …

Matching use cases

This list shows where to review this threat, rather than how high the risk is.

Typical for:

For 6 further use cases, relevance depends on the system design, for example whether it generates new content from open-ended requests or generates its own text. Show these use cases

Frequently asked questions

What is Hate Speech and Discrimination?
The AI system produces content that demeans individuals or groups on the basis of protected characteristics such as origin, gender, religion, or disability, or that calls for their exclusion.
Which mitigations help against Hate Speech and Discrimination?
The catalogue lists 5 mitigations, each with control type, effect and implementation level. First among them are Content moderation filters, Bias testing and evaluation and Safety fine-tuning (RLHF).
Which EU AI Act obligations does Hate Speech and Discrimination touch?
The catalogue maps this threat to Art. 10 and Art. 55 of the EU AI Act. In the obligations compass this leads to: Ensure data and data governance and Evaluate and mitigate systemic risks. Whether an obligation applies depends on role, risk class and deadline.

Assess this threat in your own system

The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.

Start the live demo See the free modules

Cite this entry

For reports, policies or internal documents; the link leads directly to this entry.

“Hate Speech and Discrimination”. Versatile AI Risk Assessment, AI threat catalogue, as of October 2026.
https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/hate-speech-discrimination/

← Back to the full catalogue