Versatile AI Risk Assessment

AI threat catalogue Privacy and Data Leakage Production

Sensitive Information Disclosure

An AI system discloses confidential information without anyone intending it: trade secrets, personal data, credentials or internal documents surface in the model's answers.

As of: October 2026 · Catalogue version 2026.10.24.2 · 4 mitigations · 18 verified sources

Description

Language models memorise parts of their training data and can later reproduce them verbatim or slightly altered. Confidential content reaches the answers along several paths: through memorised training data, through connected knowledge sources such as document stores and databases, or through inputs from other users. The disclosure is triggered by normal use, by deliberately crafted questions, or by prompt injection, meaning smuggled-in instructions that bypass protective filters. A second path arises in everyday work: employees enter confidential material into external AI services whose operators store the inputs and may use them for further training.

Possible impact

Trade secrets and customer data end up in the wrong hands; where personal data is involved, GDPR notification duties, fines and claims by data subjects can follow. Disclosed credentials open the door to follow-up attacks. Trust and reputation also suffer when customers learn that their data appears in answers given to third parties.

Example

Employees paste internal source code and meeting notes into a public AI chatbot to work faster. The content now sits with the provider and can flow into future model versions. One such case at a major electronics group became publicly known.

Recommended mitigations (4)

Compare effects and implementation levels. Expand each entry for the additional classification and rationale.

  • Data sanitization before training

    Technical
    Effect
    Preventive
    Implementation level
    Data
    Classification and rationale

    Primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.

  • Output filtering for sensitive patterns

    Technical
    Effect
    Preventive
    Implementation level
    Data, Application, API & agents
    Classification and rationale

    Primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.

  • PII detection and redaction

    Technical
    Effect
    Preventive, Detective
    Implementation level
    Data, Application, API & agents
    Classification and rationale

    Primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.

  • Strict access controls

    Technical
    Effect
    Preventive
    Implementation level
    Data, Application, API & agents
    Classification and rationale

    Complementary control typeOrganizational & process-based

    Primarily technical: Machine-enforced identity, permission, or scope rules constrain unauthorized access and actions; complemented by binding workflows.

Framework mappings

Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.

OWASP LLM Top 10 LLM02:2026NIST AI RMF Section 2.10 · Section 2.4 · Section 2.9 · MEASURE 2.10 · NISTAML.032 · NISTAML.035 · NISTAML.038MITRE ATLAS AML.T0057EU AI Act Article 13(1), 13(3)(b)(ii), (iv), (v) · Article 55(1)(b)GDPR Article 25(1)–(2) · EDPB Opinion 28/2024, Section 3.2BSI R2 · R25BIML BIML-LLM inference:10 · BIML-LLM raw:5 · BIML78 inference:5

Verified references (18)

Every reference states the framework, the exact location and the publishing organisation.

EU AI Act obligations related to this threat

The obligations from the obligations compass whose articles the catalogue maps to this threat. Each leads to its entry with core, action step and evidence; whether it applies to you depends on role and risk class.

  • Provide instructions for use to deployers Art. 13 · applies from Annex III: 2 December 2027 · Annex I: 2 August 2028

    Statutory roles: Provider, Product manufacturer. Classes: High-risk under Annex III, High-risk as a product or safety component.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Evaluate and mitigate systemic risks Art. 55 · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

All 62 obligations by role, risk class and deadline in the obligations compass

Terms on this page

Glossary terms that occur in this entry. Every link leads to the full explanation.

  • Prompt Injection Manipulated input or planted content redirects a language model.

More entries from the topic group Privacy and Data Leakage.

Privacy Attacks Attackers coax personal or sensitive information about the training data out of an AI model through carefully crafted queries, for example … Exfiltration from ML Application Attackers or insiders copy data, model weights or configurations straight out of a production AI application without authorisation, … Cross-Tenant Leakage (Multi-Tenant Vector DB) On AI platforms that serve several customers on shared infrastructure, one customer's data can surface in another customer's answers when …

Matching use cases

This list shows where to review this threat, rather than how high the risk is.

Particularly important for:

  • Generative AI assistant for staffStaff paste personnel records, contracts or source code for processing. If the service stores the inputs or sends them to its provider, confidential content leaves the organisation.
  • AI knowledge assistant (enterprise search)The collection contains confidential reports accessible to only some staff. If retrieval ignores permissions, the assistant reveals their contents to unauthorised users too.
  • Speech recognition and transcriptionDictation and conversations contain health, case or account information. Provider retention or overly broad sharing exposes recordings and transcripts to unauthorised people.
  • AI meeting minutes and conversation documentationSocial care recordings contain health, family and financial information. Retention by the provider or overly broad access to minutes exposes it to unauthorised people.
  • Machine translationStaff copy contracts or personnel records into a translation service. Provider retention or reuse sends confidential content outside the organisation.
  • Automated anonymisation and redactionThe system misses an unusually spelt name or information in a footnote. The document is published and the person can be identified.
  • E-discovery and technology-assisted review (TAR)The collection brings trade secrets and personnel records from many parties onto one platform. Overly broad permissions expose more information to reviewers or third parties than the case requires.
  • Online proctoringRecordings show homes, housemates and open programs. Logging or sending them to the provider exposes many people's private circumstances.
  • Emotion recognitionThe system records its attributed emotion for each recording. If these labels leak with an image or name, individuals' supposed moods, fears or intentions become known.
  • Automatic number plate recognition (ANPR)The system stores location and time for each plate. Leaked or over-retained records reveal who was where and when.
  • Clinical decision support (such as early sepsis detection)Development and operation require complete treatment histories from many patients. Unauthorised access during training or at the provider exposes diagnoses and laboratory values.
  • AI support for emergency call handlingReal call recordings go to the manufacturer for training and evaluation. Unauthorised disclosure of voices and health information causes irreversible harm.
  • Coding assistantThe assistant sends snippets from open files to the provider's service. Any credentials or customer data in them leave the organisation's environment.

Typical for:

For 4 further use cases, relevance depends on the system design, for example whether it processes personal data or processes confidential data. Show these use cases

Frequently asked questions

What is Sensitive Information Disclosure?
An AI system discloses confidential information without anyone intending it: trade secrets, personal data, credentials or internal documents surface in the model's answers.
Which mitigations help against Sensitive Information Disclosure?
The catalogue lists 4 mitigations, each with control type, effect and implementation level. First among them are Data sanitization before training, Output filtering for sensitive patterns and PII detection and redaction.
Which EU AI Act obligations does Sensitive Information Disclosure touch?
The catalogue maps this threat to Art. 13 and Art. 55 of the EU AI Act. In the obligations compass this leads to: Provide instructions for use to deployers and Evaluate and mitigate systemic risks. Whether an obligation applies depends on role, risk class and deadline.

Assess this threat in your own system

The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.

Start the live demo See the free modules

Cite this entry

For reports, policies or internal documents; the link leads directly to this entry.

“Sensitive Information Disclosure”. Versatile AI Risk Assessment, AI threat catalogue, as of October 2026.
https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/sensitive-information-disclosure/

← Back to the full catalogue