AI knowledge assistant (enterprise search)

Annotated review example: RAG assistant

Fictitious learning example. The expectations below are not measured test results or an approval. Updated: 9 October 2026.

1. Define the system boundary

   A fictitious company wants to make internal policies easier for staff to find. Its assistant answers questions from a bounded document collection and links to the passages it used. It may neither send email nor modify files. The pilot uses synthetic content only. The responsible business team names document owners, user groups and a technical operator. Before testing, the team records the model version, search index and permissions. Whether the external model service retains requests is initially unresolved. This question belongs in preparation and must be answered before real data is used. “Internal documents only” describes an origin, but does not establish a trust boundary: an internal file can also contain malicious instructions.

2. Describe the attack path and potential consequence

   A manipulated policy contains ordinary text alongside an instruction to send earlier conversation content to an outside address. Retrieval supplies the passage to the language model as context. If the model treats the document as a higher-priority instruction, this is the attack path of indirect prompt injection. The person asking the question need not be the attacker. In this bounded pilot, actual transmission is technically prevented; an answer could still be distorted or contain a harmful link. The team therefore separates three questions: was the hostile content retrieved, was its instruction followed, and could an unauthorised effect occur? Mapping this to indirect prompt injection establishes a review prompt, not a probability. Other threats, including unauthorised document access, still require their own checks.

3. Specify two mitigations

   The first mitigation is “Validation of content sources”. In this pilot, only explicitly approved repositories are indexed; origin, document version and permitted user group remain attached to each passage. Import rules detect or isolate suspicious content and log their decisions. This implementation needs testing because an allowed source can still carry hostile content. The second mitigation is “Sandbox processing of external content”. The processing service receives no sending tools or write permissions. Network access is technically restricted, and retrieved text stays marked as untrusted content. A sentence in the system prompt alone does not demonstrate this boundary. The owner documents which infrastructure rule enforces the restriction and which legitimate data flows remain necessary.

4. Define tests and evidence in advance

   The team creates synthetic documents with visible and hidden instructions, alongside normal comparison documents. Cases cover allowed and disallowed sources, different user permissions and repeated runs with the same configuration. Expected behaviour is defined beforehand: disallowed sources do not enter the index; document instructions trigger no extra tools or network connections; answers identify verifiable sources. Ordinary documents must remain useful. Each run connects a test identifier, input, retrieved passages, model and rule versions, answer and relevant network logs. A second person checks the association and records deviations. These expectations constitute a test plan, not observed results. An absent log event is meaningful only when the completeness of logging has itself been checked.

5. State limitations and prepare the next decision

   Even passing test cases would not demonstrate general safety. New document types, model changes and additional tools can alter the attack path. Before approval, the team must resolve retention by the model service, lawful basis, permission enforcement, incident handling and repeat testing. Legal classification follows intended purpose, role and class rather than the term RAG: Art. 14 concerns high-risk systems; Art. 55 addresses providers of general-purpose AI models with systemic risk. A threat reference does not automatically turn an ordinary deployer into a model provider. Preparation collects the use case, the threat, sources and outstanding evidence. It assigns no risk score and grants no operating approval. The handout continues to describe the use case; the individual review plan is exported separately from preparation. The owner also records who will resolve each outstanding question, by when, and which decision can then be made.

Three review questions for your preparation

1. Who may access which documents before retrieval?
2. Which technical boundaries prevent unauthorised actions?
3. Which versions and logs make the test reproducible?

Threat with all mitigations · Validation of content sources · Sandbox processing · OWASP LLM01:2026