Agent Skills: Resilience Patterns

Resilience Patterns

Overview

Resilience is a composition concern, not a business logic concern. Retry, timeout, backoff, and circuit-breaking belong at the workflow level, never inside the functions that do the actual work. This keeps business functions pure and testable (they return a Result), confines retry policy to one place so it never multiplies across layers, and makes failure handling explicit and auditable at the call site.

Workflows
  -> step.retry() and step.withTimeout()
  -> (resilience here)

Business Functions
  -> fn(args, deps): Result<T, E>
  -> (no retry logic here)

Infrastructure
  -> pg, redis, http
  -> (just transport)

When to Use

• Wrapping operations that can fail transiently: HTTP calls, DB reads, cache lookups, external APIs
• Setting per-attempt timeouts so nothing hangs indefinitely
• Deciding which error types are safe to retry
• Guarding non-idempotent writes (payments, creates) against accidental replay
• Protecting downstream services from retry storms with jitter and circuit breakers

When NOT to use: Inside business functions (keep them clean, see fn-args-deps), on non-idempotent writes without an idempotency key, or stacking retry at multiple layers. Pure input validation belongs in validation-boundary, not retry.

Related: result-types (what retried functions return; retry decisions key off the error variant), fn-args-deps (clean functions the workflow wraps), api-design (where Idempotency-Key and Retry-After originate), observability (tracing retried steps and timeouts).

Required Behaviors

1. Retry at Workflow Level Only

NEVER add retry logic inside business functions:

// WRONG - Retry inside function
async function getUser(args, deps) {
  let attempts = 0;
  while (attempts < 3) {
    try {
      return await deps.db.findUser(args.userId);
    } catch { attempts++; }
  }
}

// CORRECT - Clean function, workflow handles retry
async function getUser(args, deps) {
  const user = await deps.db.findUser(args.userId);
  return user ? ok(user) : err('NOT_FOUND');
}

// Workflow adds resilience
const result = await workflow(async (step) => {
  const user = await step.retry(
    () => getUser({ userId }, deps),
    { attempts: 3, backoff: 'exponential' }
  );
  return user;
});

2. Never Double-Retry

Retry at ONE level only. Multiple layers create retry explosion:

3 (API) × 3 (Service) × 3 (DB Client) = 27 attempts!

This DDoS's your own infrastructure.

3. Only Retry Transient Errors

| Error Type | Retry? | Why | |-----------|--------|-----| | TIMEOUT | Yes | Transient | | CONNECTION_ERROR | Yes | Network hiccup | | RATE_LIMITED | Yes | Wait and retry | | NOT_FOUND | NO | Resource doesn't exist | | UNAUTHORIZED | NO | Credentials wrong | | VALIDATION_FAILED | NO | Input invalid |

const data = await step.retry(
  () => fetchFromApi(),
  {
    attempts: 3,
    retryOn: (error) => {
      const retryable = ['TIMEOUT', 'CONNECTION_ERROR', 'RATE_LIMITED'];
      return retryable.includes(error);
    },
  }
);

4. Never Retry Non-Idempotent Writes

// DANGEROUS - May double-charge
await step.retry(() => chargeCard(amount), { attempts: 3 });

// SAFE - Read is idempotent
await step.retry(() => getUser(userId), { attempts: 3 });

// SAFE - With idempotency key
await step.retry(
  () => chargeCard(amount, { idempotencyKey }),
  { attempts: 3 }
);

5. Always Set Timeouts

Never let operations hang indefinitely:

const data = await step.withTimeout(
  () => slowOperation(),
  { ms: 2000 }
);

6. Always Use Jitter

Prevents thundering herd when multiple instances retry:

// Without jitter - all instances retry at same time
// With jitter - spread out, infrastructure can recover

step.retry(() => fetchData(), {
  attempts: 3,
  backoff: 'exponential',
  jitter: true,  // ALWAYS enable in production
});

7. Combine Retry and Timeout

Each attempt gets its own timeout:

const data = await step.retry(
  () => fetchData(),
  {
    attempts: 3,
    timeout: { ms: 2000 },  // 2s per attempt
  }
);
// Total max time: 3 × 2s = 6s

Recommended Defaults

| Operation | Attempts | Backoff | Initial Delay | Timeout | |-----------|----------|---------|---------------|---------| | DB read | 3 | exponential | 50ms | 5s | | DB write | 1 | - | - | 10s | | HTTP API | 3 | exponential | 100ms | 30s | | Cache | 2 | fixed | 10ms | 500ms |

Full Example

import { createWorkflow } from '@jagreehal/workflow';

// Clean business function
async function getUser(args, deps): AsyncResult<User, 'NOT_FOUND' | 'DB_ERROR'> {
  try {
    const user = await deps.db.findUser(args.userId);
    return user ? ok(user) : err('NOT_FOUND');
  } catch {
    return err('DB_ERROR');
  }
}

// Workflow adds resilience
const loadUser = createWorkflow({ getUser });

const result = await loadUser(async (step) => {
  const user = await step.retry(
    () => getUser({ userId }, deps),
    {
      attempts: 3,
      backoff: 'exponential',
      initialDelay: 100,
      maxDelay: 2000,
      jitter: true,
      timeout: { ms: 5000 },
    }
  );
  return user;
});

8. Retrying Multi-Step Operations

Sometimes you need to retry a multi-step operation. Use step.retry() to wrap the entire sequence:

const syncUserToProvider = createWorkflow({ findUser, syncUser, markSynced });

const result = await syncUserToProvider(async (step) => {
  // Retry the whole operation
  const user = await step.retry(
    async () => {
      const user = await step(() => findUser({ userId }, deps));
      await step(() => syncUser({ user }, deps));  // Must be idempotent!
      await step(() => markSynced({ userId }, deps));
      return user;
    },
    {
      attempts: 2,
      backoff: 'exponential',
    }
  );
  return user;
});

Important: The entire sequence must be idempotent. If syncUser is called twice, it should have the same effect as calling it once.

9. Circuit Breakers

When a service is down, stop hammering it. Circuit breakers prevent cascade failures:

// Circuit breaker states
// CLOSED: Normal operation, requests go through
// OPEN: Service down, fail fast without trying
// HALF_OPEN: Testing if service recovered

Circuit breakers are outside the scope of step.retry(), but consider libraries like opossum or cockatiel for production systems where dependencies fail frequently.

When to use circuit breakers:

• External APIs that may be down for extended periods
• Services with rate limits that trigger failures
• Downstream dependencies in microservices

Don't use for:

• Database calls (usually want retry instead)
• Internal function calls

10. Handling Timeout Errors

Use helpers to detect and handle timeouts:

import { isStepTimeoutError, getStepTimeoutMeta } from '@jagreehal/workflow';

const result = await workflow(async (step) => {
  const data = await step.withTimeout(
    () => slowOperation(),
    { ms: 5000 }
  );
  return data;
});

if (!result.ok && isStepTimeoutError(result.error)) {
  const meta = getStepTimeoutMeta(result.error);
  deps.logger.warn('Operation timed out', {
    timeoutMs: meta?.timeoutMs,
    attempt: meta?.attempt,
  });
}

Common Rationalizations

| Rationalization | Reality | |---|---| | "I'll just add a retry loop inside the function" | That couples business logic to failure policy and hides it from the call site. Retry at the workflow level. | | "Retrying at every layer is extra safe" | It multiplies: 3 × 3 × 3 = 27 attempts. You DDoS your own infrastructure. Retry at ONE level. | | "Retrying the charge is fine, it usually works" | A retried non-idempotent write double-charges. Only retry with an idempotency key. | | "We don't need jitter, the delay is randomized enough" | Without jitter every instance retries in lockstep, a thundering herd that prevents recovery. Always enable jitter in production. | | "A timeout will rarely trigger, skip it" | Without a timeout a hung dependency hangs your request forever and exhausts connections. Always set one. | | "Circuit breakers are for big systems" | Any external dependency that can stay down benefits. Failing fast beats hammering a dead service. |

Red Flags

• while (attempts < n) or try/catch retry loops inside business functions
• Retry configured at more than one layer for the same call
• Retrying NOT_FOUND, UNAUTHORIZED, or VALIDATION_FAILED (non-transient)
• step.retry() around a charge/create/write with no idempotency key
• jitter not enabled, or no backoff strategy set
• Any external call with no timeout
• A multi-step step.retry() block where the steps are not all idempotent

Verification

After adding resilience to an operation:

• [ ] Retry lives at the workflow level, not inside the business function
• [ ] Exactly one layer retries this call
• [ ] retryOn (or equivalent) only matches transient errors
• [ ] Non-idempotent writes either are not retried or carry an idempotency key
• [ ] Every retried/wrapped call has a per-attempt timeout
• [ ] jitter: true and a backoff strategy are set for production
• [ ] Timeout errors are detected and logged (isStepTimeoutError)
• [ ] Tests of the underlying function still pass unchanged (it stayed clean)

The Rules

| Failure Type | Where to Retry | |-------------|----------------| | Transport/network | Workflow level | | Idempotent reads | Workflow level | | Non-idempotent writes | NEVER (or with idempotency key) | | Multi-step operation | Workflow level (if idempotent) |

1. Retry at workflow level only
2. Never double-retry across layers
3. Only retry transient errors
4. Never retry non-idempotent writes without idempotency key
5. Always set timeouts
6. Always use jitter in production
7. Use circuit breakers for external services