The AI Agent Invented an Answer: A Hallucination Containment Playbook

Trigger this playbook when the agent's output is present but wrong in a way that sounds authoritative:

• It names a product, price, policy, or feature that does not exist.
• It answers using details that belong to a different user or an unrelated earlier topic.
• It confidently makes a call it has no basis for (e.g. "your photo looks valid").
• It reveals internal information it should not (team structure, internal numbers, contact details).

• On-call engineer — reproduces the bad output, traces the source, ships the guard.
• Comms lead — for SEV1, coordinates any customer correction and the privacy notification if memory blended.
• Escalation — the agent/prompt owner and, for a data leak, whoever owns the privacy/incident process.

All read-only until you reach Mitigation.

1. 1

   Reproduce Evidence

   1 — Reproduce and freeze the evidence. Pull the exact conversation and the agent's raw output. Note whether the false claim is about data (a product/price), another conversation (memory), or a judgment (something it can't perceive).

2. 2

   Check Source Data

   2 — Rule out bad or missing source data. The most common cause is underspecified data, not a "creative" model. We had a bot invent a non-existent product variant purely because the catalog data was thin; correcting the product data so it reported only the real items fixed it.

3. 3

   Check Memory Blend

   3 — Rule out memory blend. If the false detail belongs to another topic or user, per-customer memory is bleeding across contexts. We have seen per-customer memory blend unrelated topics; the fix was to scope memory and re-ask rather than assume.

4. 4

   Check Impossible Task

   4 — Rule out an impossible task. Check whether the agent was asked to judge something it cannot perceive. We had a bot escalate a damaged-item claim the instant a customer typed "I sent it" — the model cannot see whether media is real or relevant, so it should never rule on it.

5. 5

   Check Output Discipline

   5 — Check output discipline. Confirm the model is pinned for determinism and its output contract holds. We set temperature to 0 for consistent automation output, and enforce strict JSON with separate fields so downstream mapping can't invent structure. Drift here amplifies every other cause.

- Find the entity the agent invented (product / price / policy).
- Check the source of record: does the real value exist and is it complete?
- Gap or wrong value in the source  -> data cause. Mitigation A.

- Does the wrong detail match an earlier/other conversation?
- Yes  -> memory scope cause. Mitigation B.

- Is the claim a judgment about media/state the model can't observe?
- Yes  -> capability cause. Mitigation C.

• Invented entity that has a source of record → data cause (Mitigation A). Do not tune the prompt first; fix the data.
• Wrong detail traceable to another conversation → memory blend (Mitigation B). Treat as a privacy incident until proven otherwise.
• Confident judgment about unobservable state → capability cause (Mitigation C). No prompt wording makes the model able to see.
• Right facts, malformed/over-shared output → output-discipline cause (temperature, JSON contract, disclosure rules).

Ordered least → most invasive.

• A — Data: correct the source of record so only real entities exist; add a rule that the agent answers only from provided data and says it doesn't know otherwise. Re-test the exact flow.
• B — Memory: scope memory to the current topic/customer and have the agent re-ask instead of assuming; audit whether any real cross-customer leak occurred and, if so, open the privacy process.
• C — Capability: remove the impossible judgment from the flow — escalate on the claim and let a human verify out of band. Do not ask the model to assess media it can't perceive.
• D — Output discipline: set temperature to 0, enforce strict JSON with separate fields and safe defaults, and keep internal contact details and structure out of the customer-facing prompt.
• E — Guardrail pass: add an out-of-scope guard and regression-test the agent, using an LLM acting as a simulated customer to probe for the invented answer's return.

• The invented claim is a commitment a customer may act on (price, refund, availability) → SEV1, page the agent owner and loop in comms to correct it.
• A memory blend exposed another customer's data → SEV1 privacy incident; follow the privacy/notification process.
• The same invention reappears after the data/guard fix → escalate; the source of record or retrieval is deeper than one record.

1. Re-run the exact conversation that produced the false claim — confirm the agent now answers from real data or declines.
2. Run the full set of business flows in fresh chats (no carried context) with explicit pass criteria: correct branch, valid JSON, no banned words, no invented entities.
3. For a memory fix, run two unrelated conversations back to back and confirm no detail crosses over.
4. Confirm the agent no longer makes the impossible judgment and instead escalates on the claim.

Capture: the offending conversation, the raw model output, the source-of-record value, and (for memory) the earlier conversation the detail leaked from.

Comms template:

AI agent — {SEV} — hallucination contained. What: agent stated {false claim}. Cause: {thin data | memory blend | impossible judgment}. Fix: {mitigation}. Customer impact: {none | correction sent}. Guard added: {rule/test}.