Glossary

AI agents are systems that use language models to carry out multi‑step tasks on their own: they plan, call tools and interfaces, write files or trigger transactions. This significantly widens the attack surface, because manipulated input can now trigger real actions.

The catalogue treats agent risks as a threat group of its own: from tool misuse and privilege escalation to cascading failures between agents. Core controls are tightly scoped permissions, approval steps for critical actions and complete logging.

AI literacy comprises the knowledge and skills people need to use AI systems competently and assess their outputs. This includes understanding their capabilities, limitations and risks for affected people.

Tailor learning activities to prior knowledge, tasks and context of use. Keep records of training and practical exercises. The obligations compass explains the legal context of Art. 4 and identifies the document version used.

An AI model processes inputs and produces outputs such as predictions or text. A deployed AI system also includes its interface, data sources, integrations and place in human workflows. The same model can therefore have different effects and risks in different systems.

Alongside the model and version, describe who uses its output and what actions follow. Assessing a foundation model does not replace this review of the whole application.

Sources for context: NIST AI RMF Core

An AIBOM is the bill of materials for an AI system. It extends the classic software bill of materials (SBOM) by the parts that AI adds: the models in use and their version, training and fine‑tuning data, embedding and vector stores, prompt templates, and the services connected at runtime.

The benefit shows in an incident. When a weakness in a model or a dataset becomes known, the AIBOM answers which of your own systems are affected. It is also the basis for provenance evidence towards customers and supervisory authorities.

With pseudonymisation, data cannot be attributed directly to a person without separately held additional information. Anonymisation instead requires that people are no longer identifiable using means reasonably likely to be used.

Removing names alone is insufficient: combinations of text, location or events can identify people. Assess remaining identifiability in context, including embeddings and model outputs.

Sources for context: DSGVO / GDPR

A use case is a concrete task, such as reviewing documents or screening applications. An architecture pattern describes how the system solves it: for example with a predictive model, document retrieval or an agent with tools. One use case can combine several patterns.

The graphical model in this knowledge base shows a typical setup. Compare it with your actual system and record deviations; the architecture pattern alone does not determine a statutory risk class.

Sources for context: NIST AI RMF Core

Conformity assessment is a regulated procedure for checking applicable requirements. The procedure for a high‑risk AI system depends on the system and applicable legislation; not every procedure requires an external body.

A completed quick check, reading list or model card does not replace this procedure. First establish the role, intended purpose and applicable assessment route using the obligations compass.

Sources for context: EU AI Act

The context window contains the content available to a model invocation, such as instructions, conversation history and retrieved documents. Its size is usually expressed in tokens: processing units that are not equivalent to words.

Long conversations or documents may be shortened or summarised. Check whether important limitations or sources are lost. A large context window guarantees neither complete consideration nor a correct answer.

Sources for context: Context windows · Claude Docs

In data poisoning, attackers manipulate training, fine‑tuning or reference data so that a model produces deliberately wrong, biased or harmful output, often only triggered by specific input. Downstream data sources such as vector databases for retrieval pipelines are affected as well.

Countermeasures start at data provenance: signed and versioned datasets, provenance records, outlier analysis before training, and regression tests after every data update. The assessment belongs in the supply chain view of the AI system.

The degree of autonomy describes how far an AI system makes decisions and carries out actions without human involvement. It depends on the system’s capabilities as well as its configuration, permissions and approval requirements.

Greater freedom to act can increase the consequences of an incorrect decision and reduce opportunities to intervene. The task, accessible resources and technical limits also matter. Review actual permissions and control points rather than relying on the label “autonomous”.

Sources for context: NIST AI RMF Core

The threshold analysis is the screening step before a data protection impact assessment under Art. 35 GDPR: it clarifies in a structured way whether a processing operation is likely to pose a high risk to data subjects and therefore requires a full DPIA. Criteria include new technologies, extensive profiling and special categories of data.

Generative AI adds questions of its own: prompt transfers to third parties, purpose limitation for training and context data, and the traceability of automated output. A documented review path makes the decision defensible towards supervisory authorities.

An embedding model converts content such as a text passage into a numerical vector. A RAG application can use these vectors to find passages related to a question. Vector similarity is neither proof of truth nor permission to access the content.

Embeddings can contain information about their source data. Do not automatically treat them as anonymous; include their storage, access and deletion in the data review.

Sources for context: OWASP LLM08:2025

The familiar risk tiers help with orientation but are not four completely separate statutory classes. The AI Act distinguishes prohibited practices (Art. 5), high‑risk systems (Art. 6 and Annexes I/III), and specific transparency obligations (Art. 50), among other matters. Transparency obligations can apply alongside other requirements.

Review intended purpose, specific conditions, exceptions and statutory role together. A quick check provides initial orientation; record open questions and justify the subsequent classification for the actual system.

Sources for context: EU AI Act

Fine‑tuning trains an existing model further on additional examples to adapt style, terminology or task behaviour. This changes model parameters. RAG instead retrieves relevant content at runtime and supplies it as context.

A model can memorise parts of training data and later disclose them. Removing an example from a dataset does not demonstrate that its influence has been removed from the model. Assess legal basis, purpose limitation and individual rights. RAG also requires appropriate privacy and access controls.

Sources for context: NIST AI 600-1

Art. 27 of the AI Act provides for a fundamental rights impact assessment for specified deployers and uses. It considers affected people, possible harms and measures to protect their rights.

It is distinct from a data protection impact assessment but can reference relevant parts of a DPIA. Whether it is required, and under which conditions, must be assessed for the specific deployment.

Sources for context: EU AI Act

General‑purpose AI models (GPAI) are foundation models that can be used for many tasks. The EU AI Act regulates them in a dedicated chapter: providers must supply, among other things, technical documentation, information for downstream providers and a summary of training content (Art. 53).

Particularly capable models can be classified as GPAI with systemic risk. That adds obligations such as model evaluations, assessment and mitigation of systemic risks, and cybersecurity and infrastructure protection (Art. 55). For deployers, what matters most is which model category sits inside their own system.

Guardrails are the technical bounds around a model: filters on input and output, permitted and forbidden topics, checks before tool calls, budget and step limits, and fixed output formats. They act outside the model and are therefore auditable and versionable.

They do not replace a risk assessment. Filters can be bypassed, and rules that are too strict make a system unusable. The approach only becomes solid when every guardrail maps to a concrete threat, its effect is measured, and the remaining residual risk is documented.

Language models can generate plausible but incorrect facts, sources, legal provisions or programming interfaces. This is called hallucination or confabulation. Even a system with document retrieval or tool access can misrepresent sources, draw unsupported conclusions or invent information.

Check critical statements against suitable evidence and measure errors in your own use case. A citation alone does not confirm a correct answer. The required expert review also depends on the consequences of an incorrect output.

Sources for context: NIST AI 600-1

Human in the loop means people participate at specified decision or approval points. Human on the loop describes monitoring with an opportunity to intervene. Human out of the loop describes a workflow without a human decision at each individual step; overarching oversight can still exist.

These labels alone do not demonstrate effective oversight. For high‑risk systems, consider Art. 14 of the AI Act. Check that responsible people can adequately understand outputs, have enough time and authority, and can intervene when necessary.

Sources for context: EU AI Act · Art. 14

Intended purpose is the use specified by the provider, including context and conditions. The AI Act uses it when determining whether high‑risk requirements may apply. A product name alone does not sufficiently describe the purpose.

Record the task, user groups, affected people and the effect of the output. Compare the provider’s documentation with the planned use; deviations may require a new review.

Sources for context: EU AI Act

A jailbreak attempts to bypass a model’s safety rules, for example through role play, encoded input or gradual probing. The term describes the bypass objective. Prompt injection describes manipulation of instructions through input or planted content; the two can overlap.

Assess possible consequences in the specific deployment. Text output alone can cause significant harm. Connected data, tools and transactions create further possibilities for impact. Test known patterns and document the limitations of safeguards.

Sources for context: OWASP LLM01:2025

Least privilege limits access to required resources and permitted actions. For example, an assistant that reads documents does not automatically need write access or permission to send messages.

Separate reading, writing and execution. Limit permissions in time where appropriate and test revocation. Behavioural instructions in a prompt do not replace technical authorisation checks.

Sources for context: NIST SP 800-53 Rev. 5 · AC-6

The Model Context Protocol is an open standard for applications to connect tools and data sources consistently. An MCP server exposes capabilities; the client in the surrounding application mediates calls. A language model can propose a tool call but does not itself execute the protocol.

Review server provenance and integrity, authentication, permissions per tool and controls before execution. Tool descriptions and returned content can also be manipulated. Critical actions need appropriate approvals and traceable logs.

Sources for context: MCP · Architecture

Drift is the divergence between model and reality: input data shifts away from the training data, terms change meaning, usage patterns move. This can reduce the quality of results even if the system itself remains unchanged.

Drift can develop gradually and remain unnoticed. Regular measurements help detect changes early. Record reference values at release and define when the responsible people should intervene. With purchased models there is the added point that the provider can swap the model out in the background.

A model card describes a model in structured form: intended purpose, unsuitable fields of use, training data and its provenance, performance figures, known biases and limits, plus version and maintenance details.

For deployers it is the entry point to their own assessment, not a substitute for it. It answers what a model was meant for; whether it holds up in your use case is settled only by your own review. A missing model card is itself a finding for the supply chain view.

Model theft includes unauthorised access to model weights. In model extraction, attackers attempt to reproduce a model’s behaviour through systematic queries. Disclosure of training data or system prompts is a distinct issue, although one attack can combine several objectives.

Typical controls include access protection for weights, rate limits, detection of unusual query patterns and model infrastructure hardening. Also assess data leakage and system prompt disclosure as separate attack possibilities.

Sources for context: MITRE ATLAS · Modellzugriff / Model access

Monitoring examines operational signals such as errors, response quality, unusual access or model changes. Logging records events and provides data for this work. A large volume of logs alone does not establish effective monitoring.

Define thresholds, responsibilities and responses. Also check which sensitive content enters logs and how access and retention are limited.

Sources for context: NIST AI RMF Core

In a multi‑agent system several specialised agents share one task: one plans, one researches, one writes, one reviews. They exchange intermediate results and call each other, often without a human seeing the individual steps.

The risk shifts from the single output to the chain. An error or a manipulated instruction propagates and is treated by downstream agents as established groundwork. Controls belong at the handovers: checkpoints between agents, scoped permissions per role, and logging that makes the whole run reconstructable.

Non‑human identities are technical identities for services, applications and agents. For example, a service account may be assigned to an AI agent. Keys, certificates or tokens are used to authenticate the identity to other systems.

For AI systems this is a risk surface of its own. An agent needs permissions in order to act, and often keeps them longer than necessary. Clarify ownership, validity period, scope of permissions and the path to revocation. Without an inventory, no one can say after an incident what the agent was even allowed to do.

Under the GDPR, profiling uses personal data to evaluate or predict aspects of a person, such as work performance, interests or behaviour. Not every statistical analysis therefore constitutes profiling.

Separately assess whether a decision is solely automated and has significant effects. Profiling and automated decision‑making under Art. 22 GDPR are different concepts to review.

Sources for context: DSGVO / GDPR

Prompt injection refers to attacks that attempt to redirect a language model through manipulated input or planted content. Direct attacks appear in the input; indirect attacks can occur in retrieved documents, websites or emails. The objective may be to disclose confidential data or trigger unauthorised actions.

Controls combine separation of instructions from untrusted content, scoped tool permissions, input and output checks, and attack testing. No single filter guarantees protection. Record which attacks were tested and which risks remain.

Sources for context: OWASP LLM01:2025

Provenance describes the origin of content and the processing steps leading to its current state. It can include source, version, transformation, responsible party and time. A citation is a starting point; alone it proves neither authenticity nor correctness.

For RAG, keep the answer traceable to the document passages used. For review evidence, preserve the relationship between the test, system version and result.

Sources for context: W3C PROV Overview

Provider and deployer are two important roles under the AI Act; others include importers and distributors. Providers develop a system or have it developed and place it on the market or put it into service under their own name. Deployers use it under their own authority, excluding personal non‑professional use.

Determine the role for each system and activity. Under Art. 25, certain modifications or supply under one’s own name can transfer provider obligations. Not every configuration change triggers this; assess the conditions and effects in the specific case.

Sources for context: EU AI Act · Art. 3 / 25

With retrieval augmented generation, the system searches a knowledge base for relevant passages and supplies them to the model as context for its answer. This does not require retraining the model. Quality depends on the sources, retrieval and processing of the retrieved content, among other factors.

A planted document can contain instructions that the model mistakenly follows: indirect prompt injection. Separate source content from trusted instructions and enforce tool permissions technically. Retrieval must also respect the requesting person’s access rights.

Sources for context: OWASP LLM08:2025

Red teaming is the planned attempt to make your own AI system misbehave: prompt attacks, bypasses, data leakage through output, misuse of connected tools. The whole application is tested, not only the model.

To become more than a collection of anecdotes it needs repeatability: fixed attack patterns, documented results per version, and a mapping of findings to threats and controls. The EU AI Act explicitly requires evaluations including adversarial testing for models with systemic risk.

The residual risk is the risk that remains after all agreed countermeasures have been implemented. It is derived per threat from likelihood and impact. The responsible people decide whether to accept the risk, take further action or discontinue the relevant use.

A robust residual risk method documents the reasoning, the responsible role and the date for each decision. This makes it possible to trace how each risk was handled.

In this knowledge base, review candidates are threats that warrant examination because of the described system characteristics. “Review first”, “clarify” and “foundations” organise the starting points for the review. Their counts do not indicate how dangerous a specific system is.

A risk assessment still needs the actual deployment, existing controls and their effectiveness. Analysis guidance in the model can be shown or hidden; this changes neither system characteristics nor calculation rules.

Sources for context: NIST AI RMF Core

Review evidence connects the question, test conditions, system version, expected result and observed result. It shows what was tested and what can be concluded. A planned test or link to a control is not yet evidence of its effectiveness.

Also record the review scope, responsible person and unresolved issues. The reading list collects such questions for preparation; it grants neither approval nor confirmation that obligations are met.

Sources for context: NIST AI RMF Core

A sandbox restricts which files, networks and other resources a process may access. For AI agents it can limit the consequences of faulty or manipulated tool calls.

Test the boundaries actually enforced, including network destinations, file access and execution budgets. A container or test mode is not automatically an adequately isolated sandbox.

Sources for context: NIST SP 800-95

Shadow AI is the use of AI services outside governed procurement or approval, such as a private chatbot account used for work. One cause can be the need for a readily available tool.

Without an overview of services, data flows, responsibilities and contractual safeguards are difficult to verify. Clarify purpose, data types and any required agreements. An accessible approved offering and understandable usage rules can reduce ungoverned use.

Sources for context: NIST AI RMF Core

Source access permissions determine who may read a document or dataset. In a RAG application they must also apply during search, retrieval and transfer to the model. A broadly authorised service account does not by itself protect individual users.

Test both allowed and denied access with different roles. Include revoked permissions, caches and shared indexes.

Sources for context: OWASP LLM08:2025

The system boundary describes the scope of a review: for example the chat interface, model provider, document retrieval, connected tools and human approvals. External services can be part of the reviewed workflow even when another organisation operates them.

Also record the interfaces: which data leaves the system, who receives outputs and where actions can be triggered. Explain exclusions so that relevant dependencies do not disappear from the review.

Sources for context: NIST AI RMF Core

The system prompt is the leading instruction that defines a model’s role, tone, permitted topics and limits. It is usually invisible to the user, yet it shapes every answer and often carries business logic.

Two points matter for assessment. First, it is not a secret: system prompts can often be elicited or inferred, so keys and confidential rules do not belong in them. Second, it is a configuration that changes the result and should be versioned, reviewed and approved like program code.

TOMs are the safeguards with which controllers ensure the security of processing (Art. 32 GDPR): from encryption, access control and logging to training, role concepts and approval processes. They must be appropriate to the risk of the processing.

AI systems add AI‑specific measures, such as filter chains against prompt attacks, controls against data leakage through output, and monitoring of model and data changes. A risk analysis assigns every measure to a concrete threat and makes maturity measurable.

In tool use the model decides to call a function: a search, a database query, a message, a booking. It supplies the parameters, and the surrounding application performs the call.

With that the system leaves the pure text world. Every output can trigger an action, and the parameters come from text that may be manipulated. Check the permissions per tool, validation of parameters before execution, approvals for critical calls, and which calls can be undone.

A vector database enables similarity searches over embeddings. Metadata can identify document versions, provenance and access permissions. Retrieval is the lookup step; RAG combines that lookup with subsequent answer generation.

Check that retrieval returns only approved content the requesting person is allowed to access. A result is not reliable merely because it ranks highly.

Sources for context: OWASP LLM08:2025