AI threat cataloguePrivacy and Data LeakageProduction
Privacy Attacks
Attackers coax personal or sensitive information about the training data out of an AI model through carefully crafted queries, for example whether a specific person's data was used to train it.
Description
An AI model retains traces of its training data. Attackers exploit this by querying the model systematically through its normal input interface and analysing the answers statistically; they do not need to break into the IT environment. With membership inference they determine whether a specific person's data was part of the training set. With model inversion they reconstruct typical training content, such as a face from a facial recognition model. With attribute inference they derive sensitive characteristics of a person that the system was never meant to reveal. Even supposedly anonymised data sets can be linked back to specific individuals with AI support (re-identification).
Possible impact
The organisation breaches the confidentiality of personal data without any conventional intrusion, which can trigger notification duties, fines and claims by data subjects under the GDPR. Merely proving that a person was in the training data can harm them, for example with health data. Moreover, an AI model only counts as anonymous under data protection law if such attacks, carried out with reasonable means, are likely to fail; otherwise the model itself remains subject to the GDPR.
Example
A clinic trains a prediction model on patient data and offers it as an online service. Through targeted queries an attacker can prove that a specific person belonged to the study group for a rare disease, even though the model never outputs the person's name.
Recommended mitigations (5)
Every mitigation states its control type, effect, implementation level and the reason for the classification.
Differential privacy in trainingTechnical
- Effect
- Preventive
- Implementation level
- Data, Model & training
- Reason for the classification
- “Differential privacy in training” is primarily technical: Minimization, randomization, uniform processing, or query limits reduce information gained from repeated observations.
Output privacy filtersTechnical
- Effect
- Preventive
- Implementation level
- Data, Application, API & agents
- Reason for the classification
- “Output privacy filters” is primarily technical: System-enforced inspection, transformation, or blocking rules stop or neutralize disallowed content before further processing.
Query rate limitingTechnical
- Effect
- Preventive, Impact-limiting
- Implementation level
- Application, API & agents
- Complementary control type
- Organizational & process-based
- Reason for the classification
- “Query rate limiting” is primarily technical: Minimization, randomization, uniform processing, or query limits reduce information gained from repeated observations; complemented by binding workflows.
Data minimization in trainingTechnical
- Effect
- Preventive
- Implementation level
- Data, Model & training
- Reason for the classification
- “Data minimization in training” is primarily technical: Minimization, randomization, uniform processing, or query limits reduce information gained from repeated observations.
Anonymization and aggregation techniquesTechnical
- Effect
- Preventive
- Implementation level
- Data
- Reason for the classification
- “Anonymization and aggregation techniques” is primarily technical: Minimization, randomization, uniform processing, or query limits reduce information gained from repeated observations.
Framework mappings
Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.
Verified references (19)
Every reference states the framework, the exact location and the publishing organisation.
- OWASP LLM Top 10 LLM02:2025 Sensitive Information DisclosureLLM02:2025 Sensitive Information Disclosure, official category page OWASP FoundationOriginal
- NIST AI RMF Section 2.4 Data PrivacySection 2.4, pp. 7–8 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF Section 2.9 Information SecuritySection 2.9, pp. 10–11 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF MEASURE 2.10 MEASURE 2.10MEASURE 2.10, p. 30 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF NISTAML.032 ReconstructionTaxonomy Index, p. x; Section 2.4.1, pp. 28–29 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF NISTAML.033 Membership InferenceTaxonomy Index, p. x; Section 2.4.2, pp. 29–30 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF NISTAML.038 Data ExtractionTaxonomy Index, p. xi; Section 3.3.2, pp. 46–47 National Institute of Standards and Technology (NIST)Original
- MITRE ATLAS AML.T0024.000 Infer Training Data MembershipATLAS.yaml technique object with id AML.T0024.000 (pinned release v5.6.0) MITREOriginal
- MITRE ATLAS AML.T0024.001 Invert AI ModelATLAS.yaml technique object with id AML.T0024.001 (pinned release v5.6.0) MITREOriginal
- EU AI Act Article 55(1)(a) Obligations of providers of general-purpose AI models with systemic riskArticle 55(1)(a) European Union (EUR-Lex)Original
- GDPR Article 25(1)–(2) Data protection by design and by defaultArticle 25(1) and 25(2) European Union (EUR-Lex)Original
- GDPR EDPB Opinion 28/2024, Section 3.2 EDPB Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI modelsSection 3.2, especially paragraph 43 and paragraphs 44–58, pp. 14–18 (anonymity; extraction/model attacks) European Data Protection Board (EDPB)Original
- BSI R13 Re-Identifizierung von Personen aus anonymisierten Daten (Text, Bild, Video)Kap. 4, R13, p. 21 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BSI R2 Fehlende Vertraulichkeit eingegebener Daten (Text, Bild, Video)Kap. 4, R2, p. 14 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BSI R22 Rekonstruktion von Trainingsdaten (Text, Bild, Video)Kap. 4, R22, p. 28 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BSI R23 Embedding Inversion (Text, Bild, Video)Kap. 4, R23, p. 28 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BIML BIML-LLM model:6 Training Set and Prompt RevealPDF p. 18, [model:6:training set and prompt reveal] Berryville Institute of Machine Learning (BIML)Original
- BIML BIML-LLM raw:5 Data ConfidentialityPDF p. 15, [raw:5:data confidentiality] Berryville Institute of Machine Learning (BIML)Original
- BIML BIML78 raw:1 Data ConfidentialityPDF p. 10, [raw:1:data confidentiality] Berryville Institute of Machine Learning (BIML)Original
Related threats
More entries from the topic group Privacy and Data Leakage.
Assess this threat in your own system
The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.
Cite this entry
For reports, policies or internal documents; the link leads directly to this entry.
“Privacy Attacks”. Versatile AI Risk Assessment, AI threat catalogue, as of July 2026. https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/privacy-attacks/