Versatile AI Risk Assessment

AI threat catalogue Privacy and Data Leakage Production

Privacy Attacks

Attackers coax personal or sensitive information about the training data out of an AI model through carefully crafted queries, for example whether a specific person's data was used to train it.

As of: October 2026 · Catalogue version 2026.10.24.2 · 5 mitigations · 19 verified sources

Description

An AI model retains traces of its training data. Attackers exploit this by querying the model systematically through its normal input interface and analysing the answers statistically; they do not need to break into the IT environment. With membership inference they determine whether a specific person's data was part of the training set. With model inversion they reconstruct typical training content, such as a face from a facial recognition model. With attribute inference they derive sensitive characteristics of a person that the system was never meant to reveal. Even supposedly anonymised data sets can be linked back to specific individuals with AI support (re-identification).

Possible impact

The organisation breaches the confidentiality of personal data without any conventional intrusion, which can trigger notification duties, fines and claims by data subjects under the GDPR. Merely proving that a person was in the training data can harm them, for example with health data. Moreover, an AI model only counts as anonymous under data protection law if such attacks, carried out with reasonable means, are likely to fail; otherwise the model itself remains subject to the GDPR.

Example

A clinic trains a prediction model on patient data and offers it as an online service. Through targeted queries an attacker can prove that a specific person belonged to the study group for a rare disease, even though the model never outputs the person's name.

Recommended mitigations (5)

Compare effects and implementation levels. Expand each entry for the additional classification and rationale.

  • Differential privacy in training

    Technical
    Effect
    Preventive
    Implementation level
    Data, Model & training
    Classification and rationale

    Primarily technical: Minimization, randomization, uniform processing, or query limits reduce information gained from repeated observations.

  • Output privacy filters

    Technical
    Effect
    Preventive
    Implementation level
    Data, Application, API & agents
    Classification and rationale

    Primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.

  • Query rate limiting

    Technical
    Effect
    Preventive, Impact-limiting
    Implementation level
    Application, API & agents
    Classification and rationale

    Complementary control typeOrganizational & process-based

    Primarily technical: Minimization, randomization, uniform processing, or query limits reduce information gained from repeated observations; complemented by binding workflows.

  • Data minimization in training

    Technical
    Effect
    Preventive
    Implementation level
    Data, Model & training
    Classification and rationale

    Primarily technical: Minimization, randomization, uniform processing, or query limits reduce information gained from repeated observations.

  • Anonymization and aggregation techniques

    Technical
    Effect
    Preventive
    Implementation level
    Data
    Classification and rationale

    Primarily technical: Minimization, randomization, uniform processing, or query limits reduce information gained from repeated observations.

Framework mappings

Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.

OWASP LLM Top 10 LLM02:2026NIST AI RMF Section 2.4 · Section 2.9 · MEASURE 2.10 · NISTAML.032 · NISTAML.033 · NISTAML.038MITRE ATLAS AML.T0024.000 · AML.T0024.001EU AI Act Article 55(1)(a)GDPR Article 25(1)–(2) · EDPB Opinion 28/2024, Section 3.2BSI R13 · R2 · R22 · R23BIML BIML-LLM model:6 · BIML-LLM raw:5 · BIML78 raw:1

Verified references (19)

Every reference states the framework, the exact location and the publishing organisation.

EU AI Act obligations related to this threat

The obligations from the obligations compass whose articles the catalogue maps to this threat. Each leads to its entry with core, action step and evidence; whether it applies to you depends on role and risk class.

  • Evaluate and mitigate systemic risks Art. 55 · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

All 62 obligations by role, risk class and deadline in the obligations compass

More entries from the topic group Privacy and Data Leakage.

Sensitive Information Disclosure An AI system discloses confidential information without anyone intending it: trade secrets, personal data, credentials or internal … Exfiltration from ML Application Attackers or insiders copy data, model weights or configurations straight out of a production AI application without authorisation, … Cross-Tenant Leakage (Multi-Tenant Vector DB) On AI platforms that serve several customers on shared infrastructure, one customer's data can surface in another customer's answers when …

Matching use cases

This list shows where to review this threat, rather than how high the risk is.

Particularly important for:

Typical for:

For 4 further use cases, relevance depends on the system design, for example whether it processes personal data or processes confidential data. Show these use cases

Frequently asked questions

What is Privacy Attacks?
Attackers coax personal or sensitive information about the training data out of an AI model through carefully crafted queries, for example whether a specific person's data was used to train it.
Which mitigations help against Privacy Attacks?
The catalogue lists 5 mitigations, each with control type, effect and implementation level. First among them are Differential privacy in training, Output privacy filters and Query rate limiting.
Which EU AI Act obligations does Privacy Attacks touch?
The catalogue maps this threat to Art. 55 of the EU AI Act. In the obligations compass this leads to: Evaluate and mitigate systemic risks. Whether an obligation applies depends on role, risk class and deadline.

Assess this threat in your own system

The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.

Start the live demo See the free modules

Cite this entry

For reports, policies or internal documents; the link leads directly to this entry.

“Privacy Attacks”. Versatile AI Risk Assessment, AI threat catalogue, as of October 2026.
https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/privacy-attacks/

← Back to the full catalogue