Silent Automation Outage Triage Playbook

An automation that used to produce output has gone quiet, and nothing is throwing an obvious error. This playbook takes a stressed on-call from symptom to root cause with read-only diagnostics first, then a least-to-most-invasive fix menu. Work top to bottom; do not jump to restarts before you have read the evidence.

Use this playbook when a previously working automation silently stops producing output and no alert fired, or a downstream customer reports missing messages, emails, bookings, or reports.

Silent is the key word. The pipeline reports success, a queue drains, or a monitor stays green, yet the expected result never lands. Common triggers:

• A CRM stops sending email even though sends show as processed [1029].
• Outgoing messages fail and some users go missing from a sub-account [238].
• A voice or booking agent returns inconsistent or stale answers [199].
• Every workflow using a Code node halts at once, instance-wide [2312].
• Replies degrade in quality without any hard failure [1526].

Classify severity by blast radius first, then by customer visibility.

A single shared task runner is a whole-instance single point of failure: when it goes down, every workflow with a Code node times out and halts [2312]. Treat any instance-wide symptom as SEV-1 until proven narrower.

Impact is not just "is it erroring." A send marked processed that never arrives is still a full outage to the customer [1029].

Assign these before you touch anything. Keep names generic; fill with whoever is on rotation.

• Incident owner (on-call): runs the triage tree, decides mitigations, owns the timeline.
• Comms lead: posts status updates, fields customer-facing questions, shields the owner.
• Escalation contact: platform/infra owner for self-hosting, memory, and instance restarts.
• Integration owner: subject expert for the specific CRM, telephony, or workflow platform involved.

For a SEV-3 the owner may hold every role. For SEV-1, split comms out immediately.

Run diagnostics in this order. Everything in this section is read-only or state-safe. Do not restart, clear, or re-run anything until the decision tree points you there.

1. Confirm the trigger actually fired. Read the live executions for the workflow and verify an execution started when it should have [1526].
2. Read the whole pipeline, not just the model or the first node. A correct agent can still ship bad output when a downstream post-processing node drops a required field or a JSON-parser node rewrites the answer [1526].
3. Reconcile the automation tracker against live workflows in BOTH Zapier and n8n. A master tracker cross-referenced against both platforms surfaces stale drafts and inactive automations [1068].
4. Check for a toggled-off or unpublished automation. Look for an automation switched OFF by a plan downgrade and any unpublished "(Copy)" draft that never went live [1068].
5. Check for duplicate or stale paths. If two implementations run at once, a later fix may hit only one and they diverge [199].
6. For missing email, check the suppression list. A CRM can report a send as processed while the address is suppressed and receives nothing [1029].
7. For missing messages or users, scope the failure. Confirm which channel fails and whether accounts are actually missing from the sub-account [238].
8. Check runner and instance health last, because it is the most disruptive to inspect. Look for a task-runner outage halting every Code-node workflow [2312] and for memory near the plan ceiling while processing thousands of items [2460].

Read-only diagnostic checklist:

[ ] Live executions read for the affected workflow (did the trigger fire?)
[ ] Full node chain traced end-to-end (validator / JSON-parser / post-processing)
[ ] Tracker reconciled vs live workflows in Zapier AND n8n
[ ] Any automation OFF from a plan downgrade?
[ ] Any unpublished "(Copy)" draft that never went live?
[ ] Duplicate/legacy path still active alongside the new one?
[ ] CRM suppression list reviewed for the affected addresses
[ ] Correct channel and account membership confirmed
[ ] Task-runner status: up or down? Do all Code-node workflows halt?
[ ] Memory usage vs plan ceiling while processing large batches

Branch on what the read-only pass showed.

• If the monitor itself went silent, suspect it shares the failure domain of the thing it watches. A runner-down monitor that depends on the runner it watches goes quiet during the outage [2313].
• If the symptom is instance-wide and every Code-node workflow halts, it is a runner outage [2312].
• If outages recur and memory sits near the plan ceiling while processing thousands of items, it is a memory-ceiling breach, not a retry problem [2460].
• If a send shows "processed" but nothing arrives, it is suppression [1029].
• If answers are inconsistent between runs of the same request, suspect two active paths that have diverged [199].
• If the trigger fired and the model was correct but output is degraded, a downstream post-processing node is the culprit [1526].
• If a workflow simply never runs, check for a toggled-off automation or an unpublished draft [1068].

Apply the least invasive fix that resolves the confirmed root cause. Escalate up the list only when the cheaper fix does not apply.

1. Re-enable a toggled-off automation or publish the correct draft that never went live [1068].
2. Clear or review the suppression list so a suppressed address can receive email again [1029].
3. Retire the stale path fully and apply the fix to every active path so implementations cannot diverge again [199].
4. After a runner restart, re-run the halted executions so affected work completes [2312].
5. Move critical logic off Code nodes to native nodes so the integration survives a runner outage: HMAC signing to native Crypto nodes, SOAP bodies to native Set nodes [2437].
6. Rebuild the monitor as a standalone Error-Trigger workflow with no Code-node dependency, so the alert fires even when the runner is down [2313].
7. Self-host the instance for memory headroom with auto-restart and self-healing. This is the durable fix for recurring memory-ceiling outages, not more retries [2460].

Escalate the moment a cheaper fix does not fit the evidence.

• Escalate to the platform/infra owner when the symptom is instance-wide, all Code-node workflows halt, or memory sits at the plan ceiling. These need a restart or a self-host decision [2312][2460].
• Escalate to the integration owner when a critical integration keeps breaking on runner outages and needs its Code nodes replaced with native nodes [2437].
• Page the escalation contact if the alert path itself is silent, because your monitoring is inside the failure domain [2313].

Rollback criteria: if a mitigation makes output worse or widens the blast radius, revert it and drop back to the last read-only-confirmed state before trying the next item.

Verify from the customer's side, not just the dashboard.

1. Confirm the customer-visible result actually lands: the email arrives, the message sends, the booking returns a consistent answer, the report generates.
2. After a runner restart, re-run affected executions and validate the runner with a Code-node test before declaring recovery [2312].
3. For suppression fixes, send a test to a previously suppressed address and confirm delivery, not just a "processed" status [1029].
4. For divergent paths, exercise the same request twice and confirm identical answers [199].
5. For degraded output, re-run through the full node chain and confirm the required fields survive to the end [1526].

Capture evidence while it is fresh, before restarts overwrite it.

• Live-execution logs for the affected workflow [1526].
• The tracker diff: which automations were OFF, stale, or unpublished [1068].
• Memory metrics against the plan ceiling at the time of the incident [2460].
• Suppression-list state before and after any clearing [1029].

Status-update template:

[SEV-x] <automation/customer> — <UPDATE>
Impact: <who/what is affected, customer-visible symptom>
Root cause: <confirmed / suspected — e.g. runner outage, suppression, stale path>
Action: <read-only diagnostics done / mitigation applied>
Next update: <time>

Book a short retro and turn the root cause into a preventative action.

• Move any monitor outside the failure domain of what it watches; rebuild runner-dependent monitors as standalone Error-Triggers [2313].
• Replace Code nodes on critical paths with native nodes so outages cannot halt them [2437].
• Keep tracker hygiene: reconcile the master tracker against both Zapier and n8n on a schedule so stale drafts and OFF automations surface early [1068].
• Retire legacy paths fully instead of leaving them running alongside the new one [199].
• Where a step must be repeatable, run the GPT step at temperature 0 for consistent, deterministic structured output [213].
• If outages trace to memory, plan the self-host migration for memory headroom and self-healing rather than adding retries [2460].