Versatile AI Risk Assessment

AI threat catalogue Supply Chain and Provenance Supply Chain

Supply Chain – Models

Pre-trained AI models from external sources can be tampered with, carrying hidden malicious code or built-in backdoors. Anyone who adopts such a model imports the compromise straight into their own systems.

As of: October 2026 · Catalogue version 2026.10.24.2 · 4 mitigations · 17 verified sources

Description

Many organizations do not train AI models themselves but adopt pre-trained models from public platforms such as Hugging Face or from service providers. Attackers upload manipulated models there, swap out legitimate ones, or alter the model weights, the learned internal values of a model. Unsafe storage formats such as pickle open a particular attack path: merely loading such a model can execute hidden program code (a serialization attack). Built-in backdoors that only activate on specific inputs can even survive your own follow-up training. In normal operation the model appears unremarkable, so the manipulation is hard to detect without targeted checks.

Possible impact

A manipulated model can produce wrong or deliberately skewed results that the business then bases decisions on. Embedded malicious code additionally lets attackers take over systems and extract data. The organization risks operational disruption, breaches of documentation and cybersecurity duties under the EU AI Act, and a loss of trust if customers or business partners are affected by the compromise.

Example

Security researchers uploaded a well-known open-source language model in slightly altered form to a public model platform: it spread a specific piece of false information in response to certain questions while otherwise behaving completely normally. A company adopting this model without verification would hardly have noticed the manipulation.

Recommended mitigations (4)

Compare effects and implementation levels. Expand each entry for the additional classification and rationale.

  • Model signature verification

    Technical
    Effect
    Preventive, Detective
    Implementation level
    Model & training, Supply chain
    Classification and rationale

    Complementary control typeOrganizational & process-based

    Primarily technical: Cryptographic or machine-verifiable properties protect confidentiality, integrity, or provenance; complemented by binding workflows.

  • Use trusted model repositories

    Organizational & process-based
    Effect
    Preventive
    Implementation level
    Model & training, Supply chain
    Classification and rationale

    Complementary control typeTechnical

    Primarily organizational and process-based: Defined selection, operating, or lifecycle procedures make the control binding and repeatable; complemented by technical implementation.

  • Scan models for malicious code

    Technical
    Effect
    Detective
    Implementation level
    Model & training
    Classification and rationale

    Primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.

  • Use safe serialization formats (e.g., safetensors)

    Technical
    Effect
    Preventive
    Implementation level
    Model & training
    Classification and rationale

    Primarily technical: Safe formats, restrictive defaults, or protective layers reduce unsafe execution paths and exploitable attack surface.

Framework mappings

Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.

OWASP LLM Top 10 LLM04:2026 · LLM05:2026NIST AI RMF Section 2.12 · GOVERN 6.1 · MAP 4.1 · NISTAML.026 · NISTAML.051MITRE ATLAS AML.T0010 · AML.T0115.001EU AI Act Article 25(4) · Article 53(1)(a) · Article 55(1)(d)BSI R1 · R19BIML BIML-LLM model:4 · BIML78 alg:11 · BIML78 data:2

Verified references (17)

Every reference states the framework, the exact location and the publishing organisation.

EU AI Act obligations related to this threat

The obligations from the obligations compass whose articles the catalogue maps to this threat. Each leads to its entry with core, action step and evidence; whether it applies to you depends on role and risk class.

  • Support downstream providers Art. 25 Abs. 2, Art. 25 Abs. 4 · applies from Annex III: 2 December 2027 · Annex I: 2 August 2028 · amended by Reg. 2026/1744

    Statutory roles: Provider, Provider of a general-purpose AI model. Classes: High-risk under Annex III, High-risk as a product or safety component.

    Regulation (EU) 2026/1744 spelled out the initial provider duty to cooperate and at the same time created an exception: it does not apply where the initial provider has clearly specified that their system must not be turned into a high-risk system. Infringements of Art. 25(2) and (4) are now expressly subject to fines.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Draw up the technical documentation of the model Art. 53 Abs. 1 Buchst. a, Anhang XI · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: General-purpose AI model, Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Provide information to downstream providers Art. 53 Abs. 1 Buchst. b, Anhang XII · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: General-purpose AI model, Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Respect copyright and summarise the training content Art. 53 Abs. 1 Buchst. c und d · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: General-purpose AI model, Model with systemic risk.

    For models released under a free and open-source licence with publicly available parameters, the obligations under points (a) and (b) do not apply, but the copyright policy and the summary do. Models with systemic risk are excluded from this relief.

    This reference is a review prompt. A threat match alone does not establish applicability.

  • Evaluate and mitigate systemic risks Art. 55 · applies from 2 August 2025

    Statutory roles: Provider of a general-purpose AI model. Classes: Model with systemic risk.

    This reference is a review prompt. A threat match alone does not establish applicability.

All 62 obligations by role, risk class and deadline in the obligations compass

More entries from the topic group Supply Chain and Provenance.

Supply Chain – Infrastructure Attackers compromise the technical environment in which an AI system is developed and operated: cloud services, development tools, and … Supply Chain – Datasets Training and fine-tuning data from external sources can be poisoned, flawed, or legally tainted.

Matching use cases

This list shows where to review this threat, rather than how high the risk is.

Belongs on the checklist for every AI system. What to review for every AI system

Frequently asked questions

What is Supply Chain – Models?
Pre-trained AI models from external sources can be tampered with, carrying hidden malicious code or built-in backdoors. Anyone who adopts such a model imports the compromise straight into their own systems.
Which mitigations help against Supply Chain – Models?
The catalogue lists 4 mitigations, each with control type, effect and implementation level. First among them are Model signature verification, Use trusted model repositories and Scan models for malicious code.
Which EU AI Act obligations does Supply Chain – Models touch?
The catalogue maps this threat to Art. 25 Abs. 2, Art. 53 Abs. 1 Buchst. a, Art. 53 Abs. 1 Buchst. b, Art. 53 Abs. 1 Buchst. c und d and Art. 55 of the EU AI Act. In the obligations compass this leads to: Support downstream providers, Draw up the technical documentation of the model, Provide information to downstream providers, Respect copyright and summarise the training content and Evaluate and mitigate systemic risks. Whether an obligation applies depends on role, risk class and deadline.

Assess this threat in your own system

The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.

Start the live demo See the free modules

Cite this entry

For reports, policies or internal documents; the link leads directly to this entry.

“Supply Chain – Models”. Versatile AI Risk Assessment, AI threat catalogue, as of October 2026.
https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/supply-chain-models/

← Back to the full catalogue