AI threat cataloguePrivacy and Data LeakageProduction
Exfiltration from ML Application
Attackers or insiders copy data, model weights or configurations straight out of a production AI application without authorisation, exploiting software bugs, misconfigurations or legitimate access.
Description
Unlike attacks that work through the model's answers, this threat turns the application and its infrastructure into the target. Attackers exploit application vulnerabilities, openly reachable storage and interfaces, stolen credentials or insider access to siphon off training data, stored documents, chat histories, model weights or system configurations. In addition, the model itself can become the tool: through prompt injection, meaning smuggled-in instructions, attackers make it send confidential content to an address they control. The exposure lies in live operation, where the application works with real data.
Possible impact
The threat is large-scale data loss: customer data, trade secrets and internal documents, all in one stroke. Stolen model weights mean losing expensively developed intellectual property; stolen configurations and credentials enable follow-up attacks. Where personal data is affected, GDPR notification duties and fines apply, on top of recovery costs and reputational damage.
Example
A company runs its model store in the cloud but leaves it accessible without a login. One security study found more than 8,000 such openly accessible stores (container registries) on the internet, from which over 1,000 AI models could be downloaded or even modified.
Recommended mitigations (5)
Every mitigation states its control type, effect, implementation level and the reason for the classification.
Data loss prevention (DLP)Technical
- Effect
- Preventive, Detective
- Implementation level
- Data, Application, API & agents
- Complementary control type
- Organizational & process-based
- Reason for the classification
- “Data loss prevention (DLP)” is primarily technical: Automated inspection, blocking, and redaction rules at data egress points prevent data loss; data classes, exceptions, and ownership complement implementation.
Egress traffic monitoringTechnical
- Effect
- Detective
- Implementation level
- Data, Infrastructure, Use & operations
- Complementary control type
- Organizational & process-based
- Reason for the classification
- “Egress traffic monitoring” is primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators; complemented by binding workflows.
Encryption at rest and in transitTechnical
- Effect
- Preventive, Impact-limiting
- Implementation level
- Data, Infrastructure
- Reason for the classification
- “Encryption at rest and in transit” is primarily technical: Cryptographic or machine-verifiable properties protect confidentiality, integrity, or provenance.
Least privilege accessTechnical
- Effect
- Preventive, Impact-limiting
- Implementation level
- Data, Application, API & agents
- Complementary control type
- Organizational & process-based
- Reason for the classification
- “Least privilege access” is primarily technical: Machine-enforced identity, permission, or scope rules constrain unauthorized access and actions; complemented by binding workflows.
Audit loggingTechnical
- Effect
- Detective
- Implementation level
- Application, API & agents, Use & operations
- Complementary control type
- Organizational & process-based
- Reason for the classification
- “Audit logging” is primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators; complemented by binding workflows.
Framework mappings
Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.
Verified references (17)
Every reference states the framework, the exact location and the publishing organisation.
- OWASP LLM Top 10 LLM02:2025 Sensitive Information DisclosureLLM02:2025 Sensitive Information Disclosure, official category page OWASP FoundationOriginal
- NIST AI RMF Section 2.4 Data PrivacySection 2.4, pp. 7–8 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF Section 2.9 Information SecuritySection 2.9, pp. 10–11 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF MEASURE 2.10 MEASURE 2.10MEASURE 2.10, p. 30 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF NISTAML.015 Indirect Prompt InjectionTaxonomy Index, pp. x–xi; Section 3.4, pp. 50–53; Glossary, p. 110 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF NISTAML.036 Leaking information from user interactionsTaxonomy Index, p. xi; Section 3.4.3, pp. 52–53 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF NISTAML.039 Compromising connected resourcesTaxonomy Index, p. xi; Section 3.4.3, pp. 52–53 National Institute of Standards and Technology (NIST)Original
- MITRE ATLAS AML.T0024 Exfiltration via AI Inference APIATLAS.yaml technique object with id AML.T0024 (pinned release v5.6.0) MITREOriginal
- MITRE ATLAS AML.T0025 Exfiltration via Cyber MeansATLAS.yaml technique object with id AML.T0025 (pinned release v5.6.0) MITREOriginal
- EU AI Act Article 13(1), 13(3)(b)(ii), (iv), (v) Transparency and provision of information to deployersArticle 13(1), 13(3)(b)(ii), (iv), (v) European Union (EUR-Lex)Original
- EU AI Act Article 55(1)(d) Obligations of providers of general-purpose AI models with systemic riskArticle 55(1)(d) European Union (EUR-Lex)Original
- EU AI Act Article 9(1), 9(2)(a), 9(2)(d) Risk management systemArticle 9(1), 9(2)(a), 9(2)(d), read with Article 9(3) European Union (EUR-Lex)Original
- GDPR Article 25(1)–(2) Data protection by design and by defaultArticle 25(1) and 25(2) European Union (EUR-Lex)Original
- BSI R2 Fehlende Vertraulichkeit eingegebener Daten (Text, Bild, Video)Kap. 4, R2, p. 14 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BSI R28 Indirect Prompt Injections (Text)Kap. 4, R28, p. 33 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BIML BIML78 raw:3 StoragePDF p. 10, [raw:3:storage] Berryville Institute of Machine Learning (BIML)Original
- BIML BIML78 system:8 InsiderPDF p. 26, [system:8:insider] Berryville Institute of Machine Learning (BIML)Original
Terms on this page
Glossary terms that occur in this entry. Every link leads to the full explanation.
- Prompt Injection Manipulated input or planted content redirects a language model.
Related threats
More entries from the topic group Privacy and Data Leakage.
Assess this threat in your own system
The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.
Cite this entry
For reports, policies or internal documents; the link leads directly to this entry.
“Exfiltration from ML Application”. Versatile AI Risk Assessment, AI threat catalogue, as of July 2026. https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/exfiltration-ml-application/