Glossary

AI agents are systems that use language models to carry out multi‑step tasks on their own: they plan, call tools and interfaces, write files or trigger transactions. This significantly widens the attack surface, because manipulated input can now trigger real actions.

The catalogue treats agent risks as a threat group of its own: from tool misuse and privilege escalation to cascading failures between agents. Core controls are tightly scoped permissions, approval steps for critical actions and complete logging.

AI literacy comprises the knowledge and skills people need to use AI systems competently and assess their outputs. This includes understanding their capabilities, limitations and risks for affected people.

Tailor learning activities to prior knowledge, tasks and context of use. Keep records of training and practical exercises. The obligations compass explains the legal context of Art. 4 and identifies the document version used.

An AIBOM is the bill of materials for an AI system. It extends the classic software bill of materials (SBOM) by the parts that AI adds: the models in use and their version, training and fine‑tuning data, embedding and vector stores, prompt templates, and the services connected at runtime.

The benefit shows in an incident. When a weakness in a model or a dataset becomes known, the AIBOM answers which of your own systems are affected. It is also the basis for provenance evidence towards customers and supervisory authorities.

In data poisoning, attackers manipulate training, fine‑tuning or reference data so that a model produces deliberately wrong, biased or harmful output, often only triggered by specific input. Downstream data sources such as vector databases for retrieval pipelines are affected as well.

Countermeasures start at data provenance: signed and versioned datasets, provenance records, outlier analysis before training, and regression tests after every data update. The assessment belongs in the supply chain view of the AI system.

The degree of autonomy describes how far an AI system makes its own decisions and intervenes in workflows: from pure suggestions through actions with approval to fully independent operation. It is not a property of the model but a deployment decision.

For risk assessment it is one of the strongest levers. As autonomy rises, the possible damage per wrong decision grows while the opportunity to intervene shrinks. The EU AI Act explicitly names autonomy and tool access as factors of systemic risk.

The threshold analysis is the screening step before a data protection impact assessment under Art. 35 GDPR: it clarifies in a structured way whether a processing operation is likely to pose a high risk to data subjects and therefore requires a full DPIA. Criteria include new technologies, extensive profiling and special categories of data.

Generative AI adds questions of its own: prompt transfers to third parties, purpose limitation for training and context data, and the traceability of automated output. A documented review path makes the decision defensible towards supervisory authorities.

The EU AI Act orders AI systems by risk: prohibited practices (Art. 5), high‑risk systems (including Annex III, such as employment, creditworthiness, biometrics), systems with transparency obligations (Art. 50) and systems with minimal risk. The classification decides on obligations from risk management to conformity assessment.

Document the selected risk class and the reasons for the classification, including open questions and evidence still needed. A structured questionnaire helps make the reasoning traceable.

In fine‑tuning an existing model is trained further on your own examples to adapt style, domain language or task behaviour. The knowledge then sits in the weights, unlike RAG where it is supplied at runtime.

From a data protection view this is the trickier path. Personal data in the training examples cannot simply be deleted from a model again, and access and erasure requests become hard to satisfy. Clarify the legal basis, purpose limitation, and whether RAG achieves the same benefit without this risk.

General‑purpose AI models (GPAI) are foundation models that can be used for many tasks. The EU AI Act regulates them in a dedicated chapter: providers must supply, among other things, technical documentation, information for downstream providers and a summary of training content (Art. 53).

Particularly capable models can be classified as GPAI with systemic risk. That adds obligations such as model evaluations, assessment and mitigation of systemic risks, and cybersecurity and infrastructure protection (Art. 55). For deployers, what matters most is which model category sits inside their own system.

Guardrails are the technical bounds around a model: filters on input and output, permitted and forbidden topics, checks before tool calls, budget and step limits, and fixed output formats. They act outside the model and are therefore auditable and versionable.

They do not replace a risk assessment. Filters can be bypassed, and rules that are too strict make a system unusable. The approach only becomes solid when every guardrail maps to a concrete threat, its effect is measured, and the remaining residual risk is documented.

Language models produce text that sounds plausible but can be factually wrong: invented facts, sources, legal sections or programming interfaces. These confabulations are inherent to the technology, because models continue probabilities rather than querying a knowledge base.

For risk management the individual wrong answer matters less than the process around it: expert review before use, labelling of AI‑generated content, evidence requirements for critical statements, and measuring the error rate in your own use case.

The three terms distinguish where the human sits in the process. Human in the loop means nothing happens without human approval. Human on the loop means the system acts by itself while a person observes and can intervene or stop it. Human out of the loop means the system acts in production without oversight.

The classification matters for evidence. Art. 14 of the EU AI Act requires effective human oversight for high‑risk systems, and oversight is only effective if the person understands the output, has enough time and can actually stop the system. An approval button without a basis for the decision does not meet the requirement.

A jailbreak is a deliberate attempt to defeat a model’s safety rules, for example through role play, hypothetical framing, gradual probing or encoded input. Unlike prompt injection, the input here comes from the user themselves.

For deployers the single bypass matters less than what becomes possible afterwards. As long as a jailbreak only produces text the damage stays limited; once it reaches tools, data or transactions it becomes an operational risk. Regular testing with known patterns belongs in the release of every version.

The Model Context Protocol is an open standard that lets language models access tools, files and data sources in a uniform way. Instead of building a bespoke interface per application, an MCP server announces its capabilities and the model calls them through a shared protocol.

What matters for security is mainly what such a server exposes. Every connected tool widens the model’s room for action, and the tool descriptions are themselves text that can be manipulated. Check the provenance and integrity of the servers, the permissions per tool, and whether calls are logged and approved where the effect is critical.

Drift is the divergence between model and reality: input data shifts away from the training data, terms change meaning, usage patterns move. This can reduce the quality of results even if the system itself remains unchanged.

Drift can develop gradually and remain unnoticed. Regular measurements help detect changes early. Record reference values at release and define when the responsible people should intervene. With purchased models there is the added point that the provider can swap the model out in the background.

A model card describes a model in structured form: intended purpose, unsuitable fields of use, training data and its provenance, performance figures, known biases and limits, plus version and maintenance details.

For deployers it is the entry point to their own assessment, not a substitute for it. It answers what a model was meant for; whether it holds up in your use case is settled only by your own review. A missing model card is itself a finding for the supply chain view.

Model theft covers the theft of model weights and extraction via the interface: attackers issue systematic queries to reconstruct a model’s behaviour, training data or system instructions. This affects both self‑trained models and licensed models with contractual protection duties.

Typical controls are access and rate limiting, anomaly detection on query patterns, hardening of the model infrastructure, and contractual as well as technical safeguards for weights and system prompts.

In a multi‑agent system several specialised agents share one task: one plans, one researches, one writes, one reviews. They exchange intermediate results and call each other, often without a human seeing the individual steps.

The risk shifts from the single output to the chain. An error or a manipulated instruction propagates and is treated by downstream agents as established groundwork. Controls belong at the handovers: checkpoints between agents, scoped permissions per role, and logging that makes the whole run reconstructable.

Non‑human identities are technical identities for services, applications and agents. For example, a service account may be assigned to an AI agent. Keys, certificates or tokens are used to authenticate the identity to other systems.

For AI systems this is a risk surface of its own. An agent needs permissions in order to act, and often keeps them longer than necessary. Clarify ownership, validity period, scope of permissions and the path to revocation. Without an inventory, no one can say after an incident what the agent was even allowed to do.

Prompt injection refers to attacks in which manipulated input or planted content (for example in documents, websites or emails) redirects the behaviour of a language model. The model then follows the attacker’s instructions instead of the operator’s rules, for example to exfiltrate confidential data or to bypass safeguards.

Effective controls combine input and output filtering, a strict separation of instructions and data, minimal permissions for connected tools, and testing with known attack patterns. The residual risks remain subject to documentation because no filter protects completely.

The EU AI Act mainly distinguishes two roles. Providers develop an AI system or have it developed and place it on the market under their own name. Deployers use a system under their own responsibility. Different duties attach to each role, from conformity assessment to oversight in operation.

In practice the assignment is the most common point of dispute. Anyone who substantially modifies a purchased system, offers it under their own name or changes its purpose can become a provider themselves. The role should therefore be documented and justified per system, not silently assumed.

With retrieval augmented generation the system searches its own knowledge base for matching passages before answering and hands them to the model as context. That way it answers from current in‑house documents without retraining the model.

What matters for security is that retrieved content lands in the same context window as the instructions. A planted document in the knowledge base therefore acts like an instruction to the model. Permissions matter too: retrieval must mirror the access rights of the person asking, otherwise the knowledge base becomes a data leak.

Red teaming is the planned attempt to make your own AI system misbehave: prompt attacks, bypasses, data leakage through output, misuse of connected tools. The whole application is tested, not only the model.

To become more than a collection of anecdotes it needs repeatability: fixed attack patterns, documented results per version, and a mapping of findings to threats and controls. The EU AI Act explicitly requires evaluations including adversarial testing for models with systemic risk.

The residual risk is the risk that remains after all agreed countermeasures have been implemented. It is derived per threat from likelihood and impact. The responsible people decide whether to accept the risk, take further action or discontinue the relevant use.

A robust residual risk method documents the reasoning, the responsible role and the date for each decision. This makes it possible to trace how each risk was handled.

Shadow AI is the use of AI services outside governed procurement: the chatbot quickly opened in a browser, the editor extension, the private account used for work. The motive is rarely malicious; usually it is the wish to get done faster.

The problem is invisibility. Without knowing which services are in use there is no processing agreement, no purpose limitation and no record of where input went. What usually works is an easy, approved offering plus a clear rule, not a ban on its own.

The system prompt is the leading instruction that defines a model’s role, tone, permitted topics and limits. It is usually invisible to the user, yet it shapes every answer and often carries business logic.

Two points matter for assessment. First, it is not a secret: system prompts can often be elicited or inferred, so keys and confidential rules do not belong in them. Second, it is a configuration that changes the result and should be versioned, reviewed and approved like program code.

TOMs are the safeguards with which controllers ensure the security of processing (Art. 32 GDPR): from encryption, access control and logging to training, role concepts and approval processes. They must be appropriate to the risk of the processing.

AI systems add AI‑specific measures, such as filter chains against prompt attacks, controls against data leakage through output, and monitoring of model and data changes. A risk analysis assigns every measure to a concrete threat and makes maturity measurable.

In tool use the model decides to call a function: a search, a database query, a message, a booking. It supplies the parameters, and the surrounding application performs the call.

With that the system leaves the pure text world. Every output can trigger an action, and the parameters come from text that may be manipulated. Check the permissions per tool, validation of parameters before execution, approvals for critical calls, and which calls can be undone.