Versatile AI Risk Assessment

AI threat catalogue Privacy and Data Leakage Production

Exfiltration from ML Application

Attackers or insiders copy data, model weights or configurations straight out of a production AI application without authorisation, exploiting software bugs, misconfigurations or legitimate access.

As of: October 2026 · Catalogue version 2026.10.24.2 · 5 mitigations · 17 verified sources

Description

Unlike attacks that work through the model's answers, this threat turns the application and its infrastructure into the target. Attackers exploit application vulnerabilities, openly reachable storage and interfaces, stolen credentials or insider access to siphon off training data, stored documents, chat histories, model weights or system configurations. In addition, the model itself can become the tool: through prompt injection, meaning smuggled-in instructions, attackers make it send confidential content to an address they control. The exposure lies in live operation, where the application works with real data.

Possible impact

The threat is large-scale data loss: customer data, trade secrets and internal documents, all in one stroke. Stolen model weights mean losing expensively developed intellectual property; stolen configurations and credentials enable follow-up attacks. Where personal data is affected, GDPR notification duties and fines apply, on top of recovery costs and reputational damage.

Example

A company runs its model store in the cloud but leaves it accessible without a login. One security study found more than 8,000 such openly accessible stores (container registries) on the internet, from which over 1,000 AI models could be downloaded or even modified.

Recommended mitigations (5)

Compare effects and implementation levels. Expand each entry for the additional classification and rationale.

  • Data loss prevention (DLP)

    Technical
    Effect
    Preventive, Detective
    Implementation level
    Data, Application, API & agents
    Classification and rationale

    Complementary control typeOrganizational & process-based

    Primarily technical: Automated inspection, blocking, and redaction rules at data egress points prevent data loss; data classes, exceptions, and ownership complement implementation.

  • Egress traffic monitoring

    Technical
    Effect
    Detective
    Implementation level
    Data, Infrastructure, Use & operations
    Classification and rationale

    Complementary control typeOrganizational & process-based

    Primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators; complemented by binding workflows.

  • Encryption at rest and in transit

    Technical
    Effect
    Preventive, Impact-limiting
    Implementation level
    Data, Infrastructure
    Classification and rationale

    Primarily technical: Cryptographic or machine-verifiable properties protect confidentiality, integrity, or provenance.

  • Least privilege access

    Technical
    Effect
    Preventive, Impact-limiting
    Implementation level
    Data, Application, API & agents
    Classification and rationale

    Complementary control typeOrganizational & process-based

    Primarily technical: Machine-enforced identity, permission, or scope rules constrain unauthorized access and actions; complemented by binding workflows.

  • Audit logging

    Technical
    Effect
    Detective
    Implementation level
    Application, API & agents, Use & operations
    Classification and rationale

    Complementary control typeOrganizational & process-based

    Primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators; complemented by binding workflows.

Framework mappings

Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.

OWASP LLM Top 10 LLM02:2026NIST AI RMF Section 2.4 · Section 2.9 · MEASURE 2.10 · NISTAML.015 · NISTAML.036 · NISTAML.039MITRE ATLAS AML.T0024 · AML.T0025EU AI Act Article 13(1), 13(3)(b)(ii), (iv), (v) · Article 55(1)(d) · Article 9(1), 9(2)(a), 9(2)(d)GDPR Article 25(1)–(2)BSI R2 · R28BIML BIML78 raw:3 · BIML78 system:8

Verified references (17)

Every reference states the framework, the exact location and the publishing organisation.

EU AI Act obligations related to this threat

The obligations from the obligations compass whose articles the catalogue maps to this threat. Each leads to its entry with core, action step and evidence; whether it applies to you depends on role and risk class.

  • Establish and operate a risk management system Art. 9 · applies from Annex III: 2 December 2027 · Annex I: 2 August 2028

    Statutory roles: Provider, Product manufacturer. Classes: High-risk under Annex III, High-risk as a product or safety component.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Provide instructions for use to deployers Art. 13 · applies from Annex III: 2 December 2027 · Annex I: 2 August 2028

    Statutory roles: Provider, Product manufacturer. Classes: High-risk under Annex III, High-risk as a product or safety component.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Evaluate and mitigate systemic risks Art. 55 · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

All 62 obligations by role, risk class and deadline in the obligations compass

Terms on this page

Glossary terms that occur in this entry. Every link leads to the full explanation.

  • Prompt Injection Manipulated input or planted content redirects a language model.
  • Least privilege Gives people, services and agents only the permissions their task requires.

More entries from the topic group Privacy and Data Leakage.

Privacy Attacks Attackers coax personal or sensitive information about the training data out of an AI model through carefully crafted queries, for example … Sensitive Information Disclosure An AI system discloses confidential information without anyone intending it: trade secrets, personal data, credentials or internal … Cross-Tenant Leakage (Multi-Tenant Vector DB) On AI platforms that serve several customers on shared infrastructure, one customer's data can surface in another customer's answers when …

Matching use cases

This list shows where to review this threat, rather than how high the risk is.

Particularly important for:

  • AI assistant in office applicationsInjected instructions make the assistant embed internal content in a link or image. When the application loads the image, the data reaches an external server, as happened with EchoLeak.
  • Coding agentThe agent can open network connections and execute commands. An injected instruction can make it send source code or secrets to an external address.

Typical for:

For 12 further use cases, relevance depends on the system design, for example whether it passes content to services outside your environment or processes confidential data. Show these use cases

Frequently asked questions

What is Exfiltration from ML Application?
Attackers or insiders copy data, model weights or configurations straight out of a production AI application without authorisation, exploiting software bugs, misconfigurations or legitimate access.
Which mitigations help against Exfiltration from ML Application?
The catalogue lists 5 mitigations, each with control type, effect and implementation level. First among them are Data loss prevention (DLP), Egress traffic monitoring and Encryption at rest and in transit.
Which EU AI Act obligations does Exfiltration from ML Application touch?
The catalogue maps this threat to Art. 9, Art. 13 and Art. 55 of the EU AI Act. In the obligations compass this leads to: Establish and operate a risk management system, Provide instructions for use to deployers and Evaluate and mitigate systemic risks. Whether an obligation applies depends on role, risk class and deadline.

Assess this threat in your own system

The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.

Start the live demo See the free modules

Cite this entry

For reports, policies or internal documents; the link leads directly to this entry.

“Exfiltration from ML Application”. Versatile AI Risk Assessment, AI threat catalogue, as of October 2026.
https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/exfiltration-ml-application/

← Back to the full catalogue