Hardening LLM JSON Output Contracts: A Playbook

Use this when a working AI-to-structured-output pipeline breaks at the seam:

• A downstream step fails to map fields ("field not found", "cannot read property").
• The generated document/record contains a raw placeholder or an empty section.
• The parser throws and the whole reply turns into a generic "connection error."
• Output is valid sometimes — intermittent parse failures under real inputs.

• On-call engineer — reproduces the parse failure, decides the enforcement layer, ships the guard.
• Comms lead — for SEV1/SEV2, notes the pipeline status and any re-runs needed.
• Escalation — owner of the downstream consumer (doc template, DB schema, mapping).

Read-only until Mitigation.

1. 1

   Capture raw output

   1 — Capture the raw model output. Read exactly what the model returned before any parser. Ninety percent of the time the answer is visible here: a "Sure, here is..." preamble, markdown fences, a trailing comment, or a missing field.

2. 2

   Check enforcement

   2 — Check what enforcement the platform actually supports. The right control depends on the step. Choose LLM output enforcement by what the no-code/step actually supports: native Structured Outputs where available, otherwise a JSON-only instruction plus a validator. Don't assume a capability the step doesn't have.

3. 3

   Verify field contract

   3 — Check the field contract and defaults. Malformed often means under-specified. Enforce strict JSON with separate fields and N/A defaults so every field is always present and mapping stays clean. A field the model sometimes omits should have a safe default, not an optional existence check downstream.

4. 4

   Harden parse

   4 — Check the defensive parser and retry. Even with a good prompt, harden the parse: strip fences, extract the JSON object, validate against the expected fields, and retry once on failure. We stabilized document output with JSON validation plus retry and fixing placeholder leakage; the parser must be defensive, not trusting.

5. 5

   Test render gate

   5 — Check the render-time gate. Confirm nothing unvalidated can render. We guarantee placeholder-free documents with JSON validation, retries, safe defaults, and a render-time gate — the last line that refuses to emit an artifact still holding a placeholder.

- Pull one failing execution.
- Log the raw completion string, unmodified.
- Prose/fences around the JSON     -> contract not enforced. Step 2.
- Valid JSON but a field is absent  -> missing-field/default gap. Step 3.
- Placeholder text in a field       -> placeholder leak. Step 3.

• Prose/fences wrap the JSON → enforcement layer is too weak (Step 2). Move to Structured Outputs if the step supports it; otherwise JSON-only + validator.
• A field is intermittently missing → contract gap (Step 3). Add the field with a safe default; don't patch downstream.
• Placeholder leaks into output → parser + render-gate gap (Steps 4–5). Validate and gate, don't trust the model to always fill.
• JSON valid but mapped wrong → downstream mapping, not the model. Fix the consumer's field map.

Ordered least → most invasive.

• A — Tighten the prompt contract: demand JSON-only, list every required field, specify N/A defaults, forbid prose and fences. Cheapest, but never the only guard.
• B — Raise the enforcement layer: switch to native Structured Outputs if the step supports it, so the platform guarantees shape.
• C — Add a defensive parser + single retry: strip fences, extract the object, validate fields, retry once on failure, then fall back cleanly.
• D — Add safe defaults for every field: so a missing value maps to N/A rather than crashing the consumer.
• E — Add a render-time gate: refuse to emit any document/record that still contains a placeholder or empty required field.

• Structured Outputs is unsupported on the step and a JSON-only + validator combo still fails intermittently → escalate to redesign the step (e.g. move the AI call to a platform that supports enforcement).
• The malformed output is caused by a corrupt or unexpected input row rather than the model → escalate to the data owner; we have seen a pipeline silently process a corrupt legacy row and emit empty AI output.
• Field mapping is wrong at the consumer and the schema is shared → loop in the downstream owner.

1. Re-run the previously failing input and confirm it parses and maps cleanly.
2. Run the multi-input test set — several real and edge inputs — and confirm stable, placeholder-free output.
3. Force a deliberately malformed completion (inject a preamble) and confirm the defensive parser recovers or the retry succeeds.
4. Confirm the render-time gate blocks a placeholder-carrying object end to end.

Capture: the raw pre-parser completion, the expected field contract, the parser/validator version, and the input that triggered the failure.

Comms template:

AI output contract — {SEV} — {status}. Symptom: {parse fail | placeholder leak | wrong map}. Enforcement now: {Structured Outputs | JSON-only + validator}. Guards: {defensive parser + retry + render gate}. Re-runs: {n}.

Record which layer failed (prompt, enforcement, parser, or render gate) and confirm all four are present for this pipeline — a single layer is not enough. Add the triggering input to the pipeline's test set so the same malformation can't silently return.