AI threat catalogueSupply Chain and ProvenanceSupply Chain
Supply Chain – Models
Pre-trained AI models from external sources can be tampered with, carrying hidden malicious code or built-in backdoors. Anyone who adopts such a model imports the compromise straight into their own systems.
Description
Many organizations do not train AI models themselves but adopt pre-trained models from public platforms such as Hugging Face or from service providers. Attackers upload manipulated models there, swap out legitimate ones, or alter the model weights, the learned internal values of a model. Unsafe storage formats such as pickle open a particular attack path: merely loading such a model can execute hidden program code (a serialization attack). Built-in backdoors that only activate on specific inputs can even survive your own follow-up training. In normal operation the model appears unremarkable, so the manipulation is hard to detect without targeted checks.
Possible impact
A manipulated model can produce wrong or deliberately skewed results that the business then bases decisions on. Embedded malicious code additionally lets attackers take over systems and extract data. The organization risks operational disruption, breaches of documentation and cybersecurity duties under the EU AI Act, and a loss of trust if customers or business partners are affected by the compromise.
Example
Security researchers uploaded a well-known open-source language model in slightly altered form to a public model platform: it spread a specific piece of false information in response to certain questions while otherwise behaving completely normally. A company adopting this model without verification would hardly have noticed the manipulation.
Recommended mitigations (4)
Every mitigation states its control type, effect, implementation level and the reason for the classification.
Model signature verificationTechnical
- Effect
- Preventive, Detective
- Implementation level
- Model & training, Supply chain
- Complementary control type
- Organizational & process-based
- Reason for the classification
- “Model signature verification” is primarily technical: Cryptographic or machine-verifiable properties protect confidentiality, integrity, or provenance; complemented by binding workflows.
Use trusted model repositoriesOrganizational & process-based
- Effect
- Preventive
- Implementation level
- Model & training, Supply chain
- Complementary control type
- Technical
- Reason for the classification
- “Use trusted model repositories” is primarily organizational and process-based: Defined selection, operating, or lifecycle procedures make the control binding and repeatable; complemented by technical implementation.
Scan models for malicious codeTechnical
- Effect
- Detective
- Implementation level
- Model & training
- Reason for the classification
- “Scan models for malicious code” is primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.
Use safe serialization formats (e.g., safetensors)Technical
- Effect
- Preventive
- Implementation level
- Model & training
- Reason for the classification
- “Use safe serialization formats (e.g., safetensors)” is primarily technical: Safe formats, restrictive defaults, or protective layers reduce unsafe execution paths and exploitable attack surface.
Framework mappings
Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.
Verified references (15)
Every reference states the framework, the exact location and the publishing organisation.
- OWASP LLM Top 10 LLM03:2025 Supply ChainLLM03:2025 Supply Chain, official category page OWASP FoundationOriginal
- NIST AI RMF Section 2.12 Value Chain and Component IntegrationSection 2.12, p. 12 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF GOVERN 6.1 GOVERN 6.1GOVERN 6.1, p. 24 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF MAP 4.1 MAP 4.1MAP 4.1, p. 27 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF NISTAML.026 Model PoisoningTaxonomy Index, p. x; Section 2.3.4, p. 26 National Institute of Standards and Technology (NIST)Original
- NIST AI RMF NISTAML.051 Model PoisoningTaxonomy Index, p. xi; Section 3.2.2, p. 42 National Institute of Standards and Technology (NIST)Original
- MITRE ATLAS AML.T0010 AI Supply Chain CompromiseATLAS.yaml technique object with id AML.T0010 (pinned release v5.6.0) MITREOriginal
- EU AI Act Article 25(4) Responsibilities along the AI value chainArticle 25(4) European Union (EUR-Lex)Original
- EU AI Act Article 53(1)(a) Obligations for providers of general-purpose AI modelsArticle 53(1)(a) and Annex XI European Union (EUR-Lex)Original
- EU AI Act Article 55(1)(d) Obligations of providers of general-purpose AI models with systemic riskArticle 55(1)(d) European Union (EUR-Lex)Original
- BSI R1 Abhängigkeit vom entwickelnden/betreibenden Unternehmen (Text, Bild, Video)Kap. 4, R1, p. 13 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BSI R19 Vergiftung des Modells selbst (Model/Weight Poisoning) (Text, Bild, Video)Kap. 4, R19, p. 26 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BIML BIML-LLM model:4 TrojanPDF p. 17, [model:4:Trojan] Berryville Institute of Machine Learning (BIML)Original
- BIML BIML78 alg:11 ParametersPDF p. 17, [alg:11:parameters] Berryville Institute of Machine Learning (BIML)Original
- BIML BIML78 data:2 TransferPDF p. 14, [data:2:transfer] Berryville Institute of Machine Learning (BIML)Original
Related threats
More entries from the topic group Supply Chain and Provenance.
Assess this threat in your own system
The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.
Cite this entry
For reports, policies or internal documents; the link leads directly to this entry.
“Supply Chain – Models”. Versatile AI Risk Assessment, AI threat catalogue, as of July 2026. https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/supply-chain-models/