The No-Silent-Drops Workflow Playbook

This playbook enables you to diagnose why a record never reaches a downstream system, or arrives with blank fields, and to fix the cause with the smallest possible change.

The target end state is a verified data path where every source record either:

Success criteria

Decisions

Before executing, confirm:

• Automation platform (workflow engine): hosts the workflow that moves records. Its execution history is the primary diagnostic artifact.
• Source system: emits the record. Its logs confirm whether the record was ever published.
• Downstream system (destination): receives the record. Its logs confirm whether the record was ever received and what fields were written.
• API client or query tool: used to inspect records in both systems for comparison. This may be a CLI, a database client, or the platform's built-in query interface.

No specific vendor or product is assumed. The capability required is read-only inspection of each system's record state and execution logs.

What this step does

Before touching the workflow, you need to confirm whether the source system ever emitted the record. If the source never created it, no workflow change will help. This step splits the problem space in half.

1. 1

   Verify source

   Identify the source-of-truth record. Do not use a derived export or a view. Use the primary record in the source system.
   Query the source system for the known record identifier.
   If the record exists, record its timestamp and its exact field values. Compare these values against what you expect to appear downstream.
   If the record does not exist, confirm with a second query method — for example, a direct API call rather than a search interface — to rule out a search-index lag.
   Document the result. You must know, factually, whether the record exists in the source before proceeding.

2. 2

   Document result

   Done when: you can state with certainty whether the record exists in the source system, and you have recorded the source field values for comparison.

Important considerations

• If the record does not exist in the source, the problem is upstream of the workflow. The workflow is not the cause. Investigate whatever creates the source record before returning to this playbook.
• A record that exists but has blank fields in the source is a data-quality problem at the source, not a workflow parse failure. Do not "fix" this in the workflow transformation; that would mask the source problem.

What this step does

The workflow execution history tells you whether the workflow ever received a trigger for this record. This is the decisive check that separates a missing create-trigger from a parse failure.

Actions

1. 1

   Search execution

   Open the workflow execution history for the timestamp range around the expected flow time.
   Search for any execution referencing the record identifier. Search by the identifier value itself if the platform supports it, or by the timestamp if not.
   Record the outcome of the search:

2. 2

   Capture payload

   If an execution exists, capture the raw input payload to the workflow. Do not rely on how the platform displays the payload; capture the raw values where possible.

• No execution found: the workflow was never triggered for this record. The failure is in the trigger path.
• Execution found, failed: the workflow ran and failed. Inspect the failure reason.
• Execution found, succeeded, but record is missing downstream: the workflow reported success but the record did not arrive. This is a silent-write failure, and it is the most serious case.

Important considerations

Done when: you have classified the failure into exactly one of the three cases: source-missing, trigger-missing, or downstream-incomplete.

What this step does

You need to confirm what the downstream system actually received. It is possible the record arrived but was written with blank fields, which is a different failure with a different fix.

1. 1

   Compare fields

   Query the downstream system for the record identifier. Record whether it exists and what field values it holds.
   Compare the downstream field values against the source field values captured in Step 1.
   For each field that is blank downstream, determine whether the field was blank in the source payload (parse failure at source) or was dropped by the workflow transformation (mapping failure in the workflow).

2. 2

   Confirm state

   Done when: you can state the exact downstream state — record present with correct fields, present with blank fields, or entirely absent — and you know which fields are affected.

Important considerations

• A record that exists downstream with blank fields means the trigger worked and the write worked, but the transformation dropped or failed to map fields. The fix is in the transformation, not the trigger.
• A record that does not exist downstream but whose workflow execution reported success is a silent write failure. This may indicate a conditional write path that was not taken, a destination that rejected the write, or a response that was never validated.

What this step does

The fix must match the diagnosed failure case. Do not modify the workflow "in general." Change only the component that is broken.

Actions

1. For a trigger-missing case:

1. For a transformation parse failure case:

1. For a silent write failure case:

Important considerations

• If the source field was blank to begin with, do not "fix" this in the workflow. The source is the problem.
• If you are uncertain whether a change affects other records, test against the known failing record first, then against a small sample of records that previously flowed correctly.
• Prefer the smallest credible solution. A record that needs a one-field mapping fix does not justify re-architecting the transformation.

Done when: the diagnosed failure path has been corrected, and the change is scoped to the trigger, the transformation, or the write validation — not to unrelated parts of the workflow.

What this step does

A fix is not complete until the original failing record flows through correctly. This step replays the record and verifies the destination state.

1. 1

   Re-run and verify

   Re-trigger the workflow for the known record. If the workflow supports manual re-run of a specific execution, use that. Otherwise, replay the record through the source event or a direct API call to the workflow entry point.
   Confirm the execution completes without error.
   Query the downstream system and confirm the record now exists with the expected fields populated.
   Compare the downstream field values against the source values field by field.
   If any field is still blank, do not proceed. Return to the transformation mapping and repeat the diagnosis for that specific field.

2. 2

   Confirm field match

   Done when: the originally failing record exists downstream with the expected fields populated, and the field-by-field comparison passes.

Important considerations

• Re-running the same record may create a duplicate if the workflow is not idempotent. Confirm the destination handles re-processing before re-running, or verify the destination state first and only replay if the record is absent.
• A successful re-run of one record proves the fix for that record. It does not prove the general path is healthy for all record shapes.

What this step does

A single record may be the first visible instance of a systemic pattern. Before declaring the work complete, check whether other records share the characteristics that caused this failure.

1. 1

   Detect pattern

   Define the failure signature. For example: trigger filter excludes records where the status field is empty, or the transformation drops a field when a certain source field is null.
   Query the recent execution history for executions with the same failure signature.
   Query the source for other records that match the problematic pattern — for example, records with the same missing field.
   If other records match the pattern, verify whether they also failed. If they did, re-run them through the fixed path.
   If the pattern is broad, consider adding a monitoring rule that flags records with this signature rather than treating each one individually.

2. 2

   Quantify impact

   Done when: you have quantified how many records share the failure signature, and you have either re-run the affected records or documented why backfill is not required.

Important considerations

• Do not manually re-run a large backlog without confirming idempotency. If the destination cannot handle re-processing, coordinate the replay carefully or add idempotency handling first.
• This step is a detection check, not a mandate to fix every historical record. The operational decision of whether to backfill is separate from the fix.

Functional behavior

1. Submit a new record through the normal path, not a re-run, and confirm it reaches the destination with all expected fields populated.
2. Submit a record that intentionally matches the previous failure signature — for example, a record with the field that previously caused the parse failure — and confirm it now flows correctly.

Data correctness

1. Pick a record that flowed through the fixed path and compare every downstream field against the source field-by-field. Any mismatch indicates the transformation still has a gap.

Permissions

1. Confirm the identities involved have only the access they need. The diagnostic phase required read-only access. If the fix required write access, confirm no diagnostic identity retains write permissions it no longer needs.

Failure behavior

1. Inject a deliberately invalid record — one that should fail parsing — and confirm the workflow fails loudly rather than writing a blank record.
2. Confirm the failure is visible in the execution log and, if configured, generates an alert.

Observability

1. Confirm that the execution history now captures enough detail to diagnose a similar failure without re-work. The raw input payload and the failure reason must be recorded.

Repeatability

1. Run the workflow twice for the same record, or re-run the same execution, and confirm the destination does not receive a duplicate. If the destination cannot naturally deduplicate, confirm the workflow has idempotency handling.

Production readiness

1. Confirm the workflow runs without manual intervention for new records, and that the failure path — not just the success path — behaves correctly.

Rollback

• If a fix introduces a regression, revert the specific change, not the whole workflow. The trigger, transformation, and write-validation changes are independently revertible.
• If you changed the transformation mapping, restore the previous mapping and re-run the previously failing record to confirm the regression is resolved.
• If you added a validation rule that now fails loudly for legitimate records, the fix was too strict. Relax the rule to allow the legitimate shape while still catching the original failure, then re-validate.
• If you re-ran a backlog and created duplicates downstream, confirmation of rollback depends on the destination. If the destination has a deduplication mechanism, remove the duplicates there. If it does not, you will need to identify and remove the duplicate records by correlating the re-run identifiers — coordinate this with the destination owner before attempting it.

Edge cases

• No execution found but the source record exists: the trigger is the failure point. Confirm the event type and filter as described in Step 4. Do not modify the transformation.
• Source record itself has blank fields: the problem is upstream of the workflow. Fix the source or add a source-side validation. Do not mask it downstream.
• Execution succeeded but the record is absent downstream: treat as a silent write failure. Add a response check after the write call before anything else.
• Execution failed with a transient error, such as a timeout or rate limit: retry the execution before diagnosing a structural failure. Some workflow platforms handle this automatically; if yours does not, confirm the retry policy.
• Record arrives with a timestamp or timezone mismatch: this is a transformation mapping issue, not a trigger issue. Compare the source field format against the destination expectation.
• Large payload that exceeds destination limits: the write may fail at the destination even though the transformation succeeded. Check the destination's payload limits and size validation.
• Upstream outage during the expected flow window: the workflow cannot create a record from an event that never fired. If you later re-run the source export or event replay, confirm the destination handles re-processing.