Automation Error Handling: Workflows You Can Trust

An automation you can leave running needs one property before anything else: every write it makes must be safe to repeat. That is idempotency, and until you have it, adding retries makes duplicate records more likely, not less. Backoff, jitter, dead-letter queues, alerting and kill switches all sit on top of it. Here is the full stack, in build order, with JavaScript you can adapt.

What "unattended" actually demands#

Attended automation hides these bugs. You are watching, you spot the duplicate deal, you delete it, nobody knows. Once nobody is watching, the failure behavior is the product.

Three properties let you walk away. Repeat-safe, so a re-run cannot corrupt data. Self-limiting, so a downstream outage cannot turn your workflow into a load generator. Discoverable, so failed work is findable without reading run logs.

Set timeouts before you write a single retry. Microsoft's guidance is plain: a retry strategy is only as effective as the timeouts governing each attempt. The Amazon Builders' Library picks that value from a downstream service's latency percentile, p99.9 in its example.

Idempotency: the property everything else depends on#

AWS defines an idempotent service as one that promises that "making multiple identical requests has the same effect as making a single request." At-most-once and at-least-once delivery are relatively simple in a distributed system, AWS notes; exactly-once is the hard part.

You already live with duplicates. Stripe says webhook endpoints occasionally receive the same event more than once and advises logging processed event IDs. Shopify says your app might receive the same webhook more than once, for example after a network timeout or a retry.

Microsoft's example of what breaks is the cleanest: an operation that increments a value gives the wrong answer when repeated. Now imagine that value is an invoice total.

Choosing an idempotency key — and the three that betray you#

"Use an idempotency key" is the easy half. Where the key comes from is the whole mechanism. The key must be stable across attempts, so attempt 3 is recognized as the same request as attempt 1.

Three keys people reach for first are all wrong:

• The execution or run ID. Regenerated on every retry, which is exactly when you need it to match.
• A timestamp. AWS names this explicitly as an idempotency anti-pattern, citing clock skew and multiple clients using the same timestamp.
• A random UUID generated inside the step. New on every attempt, so it never matches anything.

What works: the trigger event's own ID — a Stripe event ID, a webhook delivery ID, a message ID, a row ID. Failing that, a hash of the stable payload fields. Stripe suggests V4 UUIDs generated by the caller and warns against using sensitive data such as email addresses as keys.

Three more rules from AWS. Generate keys consistently across services, or nothing recognizes the duplicate. Record the token and run the associated mutations atomically. Expire old tokens on a TTL — Stripe prunes keys once they are at least 24 hours old, after which a reused key starts a fresh request.

Classify the failure before you retry it#

Microsoft defines transient faults as momentary connectivity loss, temporary unavailability and timeouts from a busy service — problems that typically resolve on their own. Its rule: 429 and 5xx errors are typical retry candidates, while most 4xx client errors such as 400, 401, 403 and 404 indicate problems a retry will not resolve. Temporal formalizes the other half as non-retryable errors, such as invalid input, that surface immediately regardless of policy.

That last row matters more than it looks. GitHub documents returning either a 403 or a 429 when a rate limit is exceeded, so a blanket "403 means permanent" rule mis-handles the most common recoverable failure on that API.

Your platform is already retrying underneath you#

Retries are not one knob. They are a stack: your HTTP node retries, the platform replays the failed run, and the vendor SDK inside the connector retries too.

The Amazon Builders' Library models it — when each layer retries independently, attempts multiply, three becoming nine then twenty-seven, with an example of database load increasing 243-fold. Microsoft gives the smaller version, where a retry count of three on two nested calls adds up to nine attempts against the target, and names cascading retry layers an anti-pattern.

AWS's answer is to retry at only one level. Pick the layer closest to the failure and turn the others off. Then add a budget: Microsoft's example allows a process no more than 60 retries per minute against a dependency, failing immediately once that is spent. Per-request limits alone cannot stop many concurrent requests each retrying a little and collectively burying a struggling service.

Retry strategy: capped, jittered, ceilinged#

Amazon calls the pattern capped exponential backoff — you double the wait, but stop doubling at a ceiling, because the curve otherwise produces absurd delays. Jitter then scatters retries in time instead of letting them align into a spike.

The AWS Architecture Blog gives the Full Jitter formula as sleep = random(0, min(cap, base * 2 ** attempt)), alongside Equal Jitter and Decorrelated Jitter variants. In a simulation with 100 contending clients, Full Jitter cut total call count by more than half compared with un-jittered backoff.

Two hard stops from Microsoft: never do an immediate retry more than once, and never build an endless retry mechanism, because it stops an overloaded service recovering. AWS adds that you should configure a maximum number of retries or elapsed time to avoid creating backlogs that produce metastable failures.

Rate limits are their own branch#

MDN notes that 429 is defined by RFC 6585 section 4, and that a Retry-After header may be included — so your code cannot assume it exists. Write the fallback path first.

When it is there, use it. Microsoft's guidance is that a server-provided Retry-After reflects the service's own recovery timeline and takes precedence over your client-side calculation. Slack returns it in seconds alongside its 429s.

Vendors differ enough that you must read their page, not a blog post. GitHub exposes x-ratelimit-remaining and x-ratelimit-reset, says not to retry before the reset time, and warns that continuing to make requests while limited may get your integration banned. Airtable documents 5 requests per second per base and says you must wait 30 seconds before further requests succeed.

Cheaper still: do not hit the limit. n8n documents batching on the HTTP Request node through Items per Batch and Batch Interval, plus a Loop Over Items and Wait combination. Throttling on the way out beats backing off after the fact.

Partial failure: items 1–40 when item 41 dies#

A loop processes 200 rows, dies on row 41, and the platform replays the run from row 1.

AWS documents the same trap in Lambda's SQS integration: by default an error while processing a batch makes every message visible again, including the ones that succeeded, so the function can process the same message several times. The fix is a partial batch response — turn on ReportBatchItemFailures and return a batchItemFailures list naming only the message IDs that failed. Throw an exception instead and the whole batch counts as a complete failure.

Two questions translate that to your workflow. Does a replay restart from item 1? Are items 1 to 40 safe to repeat? If the second answer is yes because every write carries a stable key, partial failure stops being a data problem and becomes a wasted-quota problem. If it is no, track per-item status in a table you own and check it before each write.

Dead-letter handling: somewhere for failed work to sit#

AWS describes a dead-letter queue as the destination for messages not processed successfully, useful because it isolates them for debugging. Microsoft frames it as deferring failed work rather than discarding it.

1. Create the store

   One table: idempotency key, payload, error, failure class, attempt count, timestamp. In SQS terms this is the DLQ.

2. Set the trip threshold

   SQS uses a redrive policy with maxReceiveCount — how many times a consumer can receive a message before it moves to the DLQ. AWS notes that setting it to 1 means a single failure moves the message. Match it to your attempt ceiling.

3. Keep it longer than the source

   AWS calls it best practice to set DLQ retention longer than the original queue's, and warns that for standard queues expiry still counts from the original enqueue timestamp. A dead letter that expires before you read it is not a dead letter.

4. Build the redrive path, then alarm on depth

   AWS supports moving messages back out via dead-letter queue redrive, and suggests a CloudWatch alarm for any message moved to a DLQ. Build both. Without the replay path, your DLQ is a graveyard.

Alerting that does not get muted#

Google's SRE book is direct: "When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts," including real ones masked by the noise. It also notes that paging a human is an expensive use of their time, and that email alerts have very limited value because they become overrun with noise.

Alert on symptoms, not every cause. SRE guidance is to spend much more effort catching symptoms than causes. "Fourteen orders have not reached the CRM in an hour" is a symptom. "HTTP node returned 503" is noise.

Never page per failed item. Page on dead-letter depth and on the age of the oldest unprocessed item. AWS makes the same point for queues, suggesting you watch ApproximateAgeOfOldestMessage, where a sharp increase signals failures are not being handled properly.

Log transient faults as warnings, not errors. That is Microsoft's advice, and the reason is alert fatigue: monitoring otherwise reads normal self-healing retries as application errors. Alert on the trend instead — a rising failure rate, more retries per operation, or operations taking longer to succeed. For formal thresholds, start with the SRE Workbook's burn-rate model and its four criteria: precision, recall, detection time, reset time.

Kill switches and the shutoff you did not design#

A kill switch is a value your workflow reads as its first step and exits on. One config row, one environment variable, one boolean. Build it before you need it, because the moment you need it you will be shipping under pressure.

A circuit breaker is the automatic version. Martin Fowler's has three states: closed and passing calls through, open and failing immediately without attempting the call, and half-open, where a trial call tests whether the dependency recovered. His example trips after five consecutive failures and resets on success; a more sophisticated breaker, he notes, might trip on a failure rate instead, say 50%.

Your platform already has a kill switch, and you did not tune it.

Shopify's version is sharper. It retries a failing HTTPS webhook 8 times over the next 4 hours, and after 8 consecutive failures the subscription is automatically deleted if it was configured through the Admin API. Your integration does not pause. It disappears.

Worked example: order webhook to CRM, written twice#

Here is the version that passes testing and fails at 2am.

// Version 1 — looks fine, is not.
export default async function handleOrder(order) {
  const contact = await crm.contacts.create({ email: order.customer.email });
  await crm.deals.create({ contactId: contact.id, amount: order.total });
  return { ok: true };
}

deals.create times out after the CRM already committed the deal. The platform replays the run. Now there are two contacts and two deals, and sales calls the same customer twice.

The hardened version changes four things: the key comes from the trigger event, failures are classified before they are retried, backoff is capped and jittered, and anything unrecoverable lands in a dead-letter store.

const MAX_ATTEMPTS = 5;
const BASE_MS = 500;
const CAP_MS = 20_000;

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

// AWS "Full Jitter": random(0, min(cap, base * 2 ** attempt))
const fullJitter = (attempt) =>
  Math.random() * Math.min(CAP_MS, BASE_MS * 2 ** attempt);

class PermanentError extends Error {}

function classify(status) {
  if (status === 429) return 'rate-limited';
  if (status >= 500) return 'transient';
  if (status >= 400) return 'permanent';
  return 'ok';
}

function waitFor(response, attempt) {
  // Retry-After MAY be present on a 429. RFC 6585 does not require it.
  const seconds = Number(response.headers.get('retry-after'));
  return Number.isFinite(seconds) && seconds > 0
    ? seconds * 1000
    : fullJitter(attempt);
}

async function post(path, body, idempotencyKey) {
  for (let attempt = 0; attempt < MAX_ATTEMPTS; attempt++) {
    let response;
    try {
      response = await fetch(`https://api.your-crm.example${path}`, {
        method: 'POST',
        signal: AbortSignal.timeout(10_000), // timeout before retry logic
        headers: {
          'content-type': 'application/json',
          'idempotency-key': idempotencyKey,
        },
        body: JSON.stringify(body),
      });
    } catch (networkOrTimeout) {
      // Outcome unknown. Safe to repeat only because of the key above.
      await sleep(fullJitter(attempt));
      continue;
    }

    const verdict = classify(response.status);
    if (verdict === 'ok') return response.json();
    if (verdict === 'permanent') {
      throw new PermanentError(`${response.status} on ${path}`);
    }
    await sleep(waitFor(response, attempt));
  }
  throw new Error(`Exhausted ${MAX_ATTEMPTS} attempts on ${path}`);
}

export async function handleOrder(event) {
  if (await killSwitchOn('order-to-crm')) return { skipped: true };

  // The key derives from the trigger event, never from this run.
  const key = `order-to-crm:${event.id}`;
  if (await alreadyProcessed(key)) return { duplicate: true };

  try {
    const contact = await post(
      '/contacts',
      { email: event.data.object.customer_email },
      `${key}:contact`
    );
    await post(
      '/deals',
      { contactId: contact.id, amount: event.data.object.amount_total },
      `${key}:deal`
    );
    await markProcessed(key);
    return { ok: true };
  } catch (err) {
    await deadLetter({
      key,
      payload: event,
      error: String(err),
      permanent: err instanceof PermanentError,
    });
    return { ok: false };
  }
}

killSwitchOn, alreadyProcessed, markProcessed and deadLetter are yours to write against whatever store you have. AWS advises recording the token and running the associated mutations atomically, so use a transaction if your store supports one.

One detail worth copying from Stripe: it saves the status code and body of the first request for a given key whether or not that request succeeded, and returns the same result to later requests using that key. A retry can therefore return a stored 500. Handle it.

Test with dummy data before production#

Microsoft's testing guidance: build a mock service returning the range of errors the real one can produce, cover every error type your strategy detects, and run high-load and concurrent tests. AWS adds that you should validate idempotency across successful, failed and duplicate requests. That is an afternoon of work:

1. Point the workflow at a sandbox account and 20 fake records, including ugly ones — missing email, emoji in the name, a 0.00 total.
2. Send the identical trigger event twice. Exactly one record should exist afterwards. If two exist, stop and fix the key.
3. Force a 500 from a mock endpoint and watch the backoff intervals in the logs. They should grow and vary.
4. Force a 429 twice — once with Retry-After and once without — and confirm both paths work.
5. Kill the run mid-loop, replay it, and check that items 1 to 40 were not duplicated.
6. Add a dead-letter row by hand and run your redrive workflow on it.

What the no-code platforms hand you#

n8n documents an error workflow that runs when a workflow fails, set in Workflow Settings and starting with an Error Trigger node. That trigger receives the error message and stack, the last node executed, and an execution.retryOf field present only on retried executions — useful for detecting a replay.

Still picking a platform? The reliability differences matter more than the trigger catalog — the n8n, Make and Zapier comparison covers where each lands. For Make, verify error-handler behavior inside your own account rather than from a blog: its help center could not be independently checked for this article, and secondhand descriptions of its error-handling directives go stale.

Who this is not for#

Skip most of this if your automation only reads, or writes something a human can trivially reverse. A workflow that applies a Gmail label, posts a digest to Slack or appends to a scratch sheet has a worst case of "someone deletes a row." Build it, watch it, move on — the inbox triage walkthrough is the right shape for that.

Skip it for one-off migrations you sit and watch, too. Attended is a legitimate control. The full stack earns its cost the moment a workflow writes to a system of record: CRM, billing, inventory, payroll, anything customer-facing.

The pre-flight checklist#

Before you walk away from an automation, all seven should be true.

1. Every write carries an idempotency key derived from the trigger event ID or a hash of stable payload fields.
2. Every outbound call has a timeout, set before any retry logic exists.
3. Failures are classified into transient, rate-limited and permanent, and only the first two are retried.
4. Retries happen at exactly one layer, with capped exponential backoff, jitter and a hard attempt ceiling.
5. Retry-After is honored when present, with a working fallback when it is not.
6. Failed work lands in a dead-letter store with a redrive path and retention longer than the source.
7. Exactly one alert fires on dead-letter depth and age, and nothing pages you per failed item.

The commitment#

If your workflow writes to a system of record, do not turn it on until items 1, 4, 6 and 7 are done. Those four are non-negotiable and take an afternoon. Items 2, 3 and 5 take a second afternoon and pay for themselves the first time a vendor has a bad hour.

If it only reads, labels or notifies, ship it today and add the rest the day it starts writing. The trigger for upgrading is not workflow complexity or run volume — it is the first irreversible action. Before the next build, run the same scrutiny over the tools themselves with an AI tool security checklist.