AI threat catalogueAttacks on the Running Model and ServiceProduction
Model Reconnaissance
Attackers gather information about a deployed model, such as its design, the model family used and its capabilities, to prepare more targeted attacks later. This is the reconnaissance phase before theft or evasion.
Description
Before a targeted attack, attackers scout out the deployed model. Through the normal interface they probe which model family and version lies behind it (fingerprinting), which architecture is likely in use, and what capabilities and output categories the model has. To do so they analyse answers, accompanying metadata, verbose error messages or publicly accessible artefacts. This reconnaissance is usually not an end in itself but the precursor to model theft or to adversarial inputs that are then tailored precisely.
Possible impact
The immediate damage is small, but the insights gained make follow-on attacks considerably more effective and harder to defend against. Once the underlying model family is identified, known weaknesses of that family can be exploited in a targeted way, especially since many services build on the same base models. At the same time, internal model and configuration details can leak as trade secrets.
Example
An attacker sends an AI service a series of systematic test questions and analyses phrasing, response times and error messages. From this they infer the base model in use and aim their next attack squarely at its known weaknesses.
Recommended mitigations (6)
Every mitigation states its control type, effect, implementation level and the reason for the classification.
Query pattern anomaly detectionTechnical
- Effect
- Detective
- Implementation level
- Application, API & agents, Infrastructure
- Reason for the classification
- “Query pattern anomaly detection” is primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.
Output randomization to prevent fingerprintingTechnical
- Effect
- Preventive
- Implementation level
- Application, API & agents, Infrastructure
- Reason for the classification
- “Output randomization to prevent fingerprinting” is primarily technical: Minimization, randomization, uniform processing, or query limits reduce information gained from repeated observations.
Minimize model metadata in responsesTechnical
- Effect
- Preventive
- Implementation level
- Application, API & agents, Infrastructure
- Reason for the classification
- “Minimize model metadata in responses” is primarily technical: Minimization, randomization, uniform processing, or query limits reduce information gained from repeated observations.
Rate limiting per sessionTechnical
- Effect
- Preventive, Impact-limiting
- Implementation level
- Application, API & agents, Infrastructure
- Complementary control type
- Organizational & process-based
- Reason for the classification
- “Rate limiting per session” is primarily technical: Minimization, randomization, uniform processing, or query limits reduce information gained from repeated observations; complemented by binding workflows.
Honeypot queries for attack detectionTechnical
- Effect
- Detective
- Implementation level
- Application, API & agents, Infrastructure, Use & operations
- Reason for the classification
- “Honeypot queries for attack detection” is primarily technical: Software or analytical tools systematically produce and evaluate measurements, deviations, or attack indicators.
Access controls and authenticationTechnical
- Effect
- Preventive
- Implementation level
- Application, API & agents, Infrastructure
- Complementary control type
- Organizational & process-based
- Reason for the classification
- “Access controls and authentication” is primarily technical: Machine-enforced identity, permission, or scope rules constrain unauthorized access and actions; complemented by binding workflows.
Framework mappings
Verified locations in OWASP, NIST AI RMF, MITRE ATLAS, the EU AI Act and further frameworks. The mappings are taxonomic, not evidence of compliance.
Verified references (12)
Every reference states the framework, the exact location and the publishing organisation.
- OWASP LLM Top 10 LLM02:2025 Sensitive Information DisclosureLLM02:2025 Sensitive Information Disclosure, official category page OWASP FoundationOriginal
- OWASP LLM Top 10 LLM10:2025 Unbounded ConsumptionLLM10:2025 Unbounded Consumption, official category page OWASP FoundationOriginal
- NIST AI RMF NISTAML.031 Model ExtractionTaxonomy Index, p. x; Section 2.4.4, pp. 31–32; Section 3.3.2, p. 47 National Institute of Standards and Technology (NIST)Original
- MITRE ATLAS AML.T0002 Acquire Public AI ArtifactsATLAS.yaml technique object with id AML.T0002 (pinned release v5.6.0) MITREOriginal
- MITRE ATLAS AML.T0014 Discover AI Model FamilyATLAS.yaml technique object with id AML.T0014 (pinned release v5.6.0) MITREOriginal
- MITRE ATLAS AML.T0040 AI Model Inference API AccessATLAS.yaml technique object with id AML.T0040 (pinned release v5.6.0) MITREOriginal
- EU AI Act Article 55(1)(d) Obligations of providers of general-purpose AI models with systemic riskArticle 55(1)(d) European Union (EUR-Lex)Original
- BSI R12 Wissenssammlung und -aufbereitung im Kontext krimineller Aktivitäten (Text, Bild)Kap. 4, R12, p. 20 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BSI R25 Extraktion von Kommunikationsdaten und hinterlegten Informationen (Text, Bild, Video)Kap. 4, R25, p. 30 Bundesamt für Sicherheit in der Informationstechnik (BSI)Original
- BIML BIML-LLM input:3 Open to the PublicPDF p. 16, [input:3:open to the public] Berryville Institute of Machine Learning (BIML)Original
- BIML BIML78 assembly:6 FilterPDF p. 13, [assembly:6:filter] Berryville Institute of Machine Learning (BIML)Original
- BIML BIML78 inference:3 Confidence ScoresPDF p. 20, [inference:3:confidence scores] Berryville Institute of Machine Learning (BIML)Original
Related threats
More entries from the topic group Attacks on the Running Model and Service.
Assess this threat in your own system
The live demo contains all 52 threats of this catalogue, including the EU AI Act and GDPR assessment. The free single modules cover AI risk, the EU AI Act and GDPR. No sign-up; the assessment runs locally in your browser.
Cite this entry
For reports, policies or internal documents; the link leads directly to this entry.
“Model Reconnaissance”. Versatile AI Risk Assessment, AI threat catalogue, as of July 2026. https://www.versatile-ai-risk-assessment.com/en/wissensbasis/threats/model-reconnaissance/