How to buildOctober 11, 202612 min read
APIs and integrationsStep 5 of 8

External APIs: timeouts, retries, rate limits and graceful degradation

In short

Every call to a third-party API needs a per-attempt timeout and an overall deadline; retry only transient failures (network, timeout, 408, 429, 5xx) and only for idempotent requests, with exponential pauses, random jitter and respect for the Retry-After header. If the service stays down, a circuit breaker stops calling it and the application answers from a cache or with an honest fallback instead of failing its own pages.

How-toLevel: IntermediateExternal APIs & Error HandlingPractice: ~1–2 h

After this you will be able to

  • set a per-attempt timeout and an overall deadline with AbortSignal.timeout and AbortSignal.any and tell a timeout from a cancellation
  • classify external API errors and retry only transient failures of idempotent requests, and POST only with an Idempotency-Key
  • implement exponential backoff with Full Jitter, respect 429 and Retry-After and limit your own pace with a token bucket
  • add a circuit breaker and a cached fallback and test everything against a local capricious server

Verified with

Node.js 24 LTS (24.18)TypeScript 7.0 (tsc 7.0.2)@types/node 24Fetch API, AbortSignal.timeout / AbortSignal.anynode:test

Checked against docs: October 11, 2026

Contents
  1. 01A third-party API can always fail
  2. 02Timeouts: per attempt and for the whole operation
  3. 03Which errors to retry and which not
  4. 04Exponential backoff and jitter
  5. 05Rate limits: 429, Retry-After and your own limiter
  6. 06Circuit breaker and graceful degradation
  7. 07Testing: a capricious server, tests and a call log

A third-party API can always fail

A payment gateway, exchange rates, a CRM, a delivery service — all of them are someone else’s servers with their own outages, deployments and limits. If code simply calls await fetch(url), a hung service holds your request and connection, a burst of 503s turns into errors on your pages, and naive instant retries finish off a service that is trying to recover. The task is to decide up front, for every call, how long to wait, what to retry and what to show when there is no answer.

No connection, connection reset

What it looks like in Node.js
fetch rejects with a TypeError
What we do
Retry after a pause if the request is idempotent

Service hangs

What it looks like in Node.js
No response; with a timeout — TimeoutError
What we do
Per-attempt timeout, then retry

Transient server error

What it looks like in Node.js
500, 502, 503, 504, 408
What we do
Retry with an exponential pause

Rate limit exceeded

What it looks like in Node.js
429, often with Retry-After
What we do
Wait as long as the server asks or hand the job to the background

Error in our request

What it looks like in Node.js
400, 401, 403, 404, 422
What we do
Do not retry: the answer will not change

Service down for a long time

What it looks like in Node.js
Nothing but failures in a row
What we do
Circuit breaker and degradation: cache or fallback
  1. 01Application serviceCache and degradation: what to show when there is no answer
  2. 02Circuit breakerDo not call a service that is down
  3. 03Rate limiterStay under the provider limit
  4. 04fetchWithRetryTimeouts, error classification, backoff, Retry-After
  5. 05External APISomeone else’s server with its own outages and limits
The protection layers in this guide, top to bottom along the request path.

This guide is for backend developers who are comfortable with async/await and error handling. The code is tested on Node.js 24.18 and TypeScript 7.0; there are no runtime dependencies, only the built-in fetch, AbortSignal and node:test. Node.js 24 runs .ts files directly: type stripping is stable since 24.12, and tsc checks the types. This guide covers the mechanics of a reliable call; a typed client for a specific API with response schemas is a separate recipe.

bash
mkdir resilient-fetch
cd resilient-fetch
npm init -y
npm pkg set type=module
npm pkg set private=true --json
npm pkg delete main scripts.test
npm pkg set scripts.test="node --test" scripts.typecheck="tsc"
npm install -D typescript @types/node@24
mkdir src test
Project: ES modules, TypeScript only for type checking
tsconfig.json
{
  "compilerOptions": {
    "target": "es2024",
    "module": "nodenext",
    "strict": true,
    "noEmit": true,
    "allowImportingTsExtensions": true,
    "erasableSyntaxOnly": true,
    "verbatimModuleSyntax": true,
    "skipLibCheck": true,
    "types": ["node"]
  },
  "include": ["src", "test"]
}
tsc settings: erasable syntax only, imports with the .ts extension

Timeouts: per attempt and for the whole operation

You need two timeouts. The attempt timeout cuts off one hung request and gives a retry a chance. The overall deadline limits the whole operation including retries and pauses: the user or the calling service will not wait longer than that, however many attempts remain. Both are standard signals: AbortSignal.timeout(ms) fires on its own after the given time, and AbortSignal.any([...]) combines it with a cancellation signal from the caller. According to MDN, a timeout aborts with a TimeoutError while a manual abort gives an AbortError — this is how the wrapper tells “the server was too slow” from “we were cancelled”.

How to pick the number. The AWS Builders Library article on timeouts advises starting from the latency metrics of the downstream service: choose an acceptable rate of false timeouts, for example 0.1%, and take the matching percentile — p99.9. If there are no metrics yet, in our experience 2–5 seconds per attempt is a reasonable start for a typical JSON API, as long as you record the duration of every call and tune the value on real data.

In the wrapper below every attempt gets AbortSignal.any([overall, AbortSignal.timeout(attemptTimeoutMs)]), where overall is the deadline plus the caller’s cancellation. The pause between attempts listens to overall too, so the deadline does not “oversleep” a long pause.

Which errors to retry and which not

A retry only helps against transient failures. A 400 or 401 for the same request will not change no matter how often you repeat it — that is an error in the data or the key, and it must be surfaced and fixed. The second question is whether a retry is safe at all. RFC 9110 defines idempotent methods — PUT, DELETE and the safe GET, HEAD, OPTIONS: several identical requests have the same effect as one, which is exactly why they can be repeated after a communication failure. For non-idempotent methods the standard says a client SHOULD NOT retry them automatically unless it knows the request is actually idempotent. AWS puts it shorter: APIs with side effects are not safe to retry unless they provide idempotency.

Network error, connection reset

Retry?
Yes, if the method is idempotent or there is a key
Comment
The request may have reached the server before the reset

Attempt timeout

Retry?
Yes, under the same condition
Comment
The server may have performed the action

408 Request Timeout

Retry?
Yes
Comment
RFC 9110 allows the client to repeat the request

429 Too Many Requests

Retry?
Yes, after Retry-After
Comment
If the wait is longer than allowed, hand off to the background

500, 502, 503, 504

Retry?
Yes, with an exponential pause
Comment
A 503 may also come with Retry-After

Other 4xx

Retry?
No
Comment
The response is returned to the caller as is

Cancelled by the caller

Retry?
Never
Comment
The user has left — nobody needs the answer

For POST the wrapper retries only when the headers contain Idempotency-Key: a server that supports such a key recognises the repeat and returns the original result instead of creating a second record. The server side of this is covered in the API design guide; the key is generated once per operation and is the same in every attempt.

A failure of the external service after all attempts is expressed by a dedicated error class. This matters for the next layers: the circuit breaker and the cache react only to UpstreamError, while a 4xx about our own request must neither trip the breaker nor hide behind a stale cache.

src/errors.ts
export type FailureKind = "network" | "timeout" | "deadline" | "http";

// The external service failed: network, timeout or 408/429/5xx after all attempts.
// A 4xx about our own request is not an UpstreamError: the response is returned as is.
export class UpstreamError extends Error {
  readonly kind: FailureKind;
  readonly status: number | undefined;
  readonly attempts: number;
  readonly retryAfterMs: number | undefined;

  constructor(
    message: string,
    details: { kind: FailureKind; attempts: number; status?: number; retryAfterMs?: number; cause?: unknown },
  ) {
    super(message, { cause: details.cause });
    this.name = "UpstreamError";
    this.kind = details.kind;
    this.status = details.status;
    this.attempts = details.attempts;
    this.retryAfterMs = details.retryAfterMs;
  }
}

export class CircuitOpenError extends Error {
  constructor(name: string) {
    super(`Circuit "${name}" is open: the call was not made`);
    this.name = "CircuitOpenError";
  }
}
Two error classes: an external service failure and an open circuit

Exponential backoff and jitter

Retrying immediately is pointless: a service that has just answered 503 will most likely answer the same a millisecond later. Exponential backoff doubles the pause after every failure, and a cap keeps it from growing forever. But if an outage hits thousands of clients at once, their retries come back in synchronised waves. The AWS Architecture Blog post “Exponential Backoff And Jitter” (Marc Brooker) shows this with a simulation and compares variants of random spread — jitter:

Capped exponential

Formula from the AWS post
sleep = min(cap, base * 2 ** attempt)
What the post says
Clients keep contending in waves

Full Jitter

Formula from the AWS post
sleep = random_between(0, min(cap, base * 2 ** attempt))
What the post says
Least work for the server, slightly more time

Equal Jitter

Formula from the AWS post
temp = min(cap, base * 2 ** attempt); sleep = temp / 2 + random_between(0, temp / 2)
What the post says
The worst of the jittered variants

Decorrelated Jitter

Formula from the AWS post
sleep = min(cap, random_between(base, sleep * 3))
What the post says
The choice between it and Full Jitter is, in the author’s words, “less clear”

The post concludes that jitter should be considered a standard approach for remote clients. We use Full Jitter — it is the simplest and causes the fewest extra calls. With base = 200 ms and cap = 5000 ms the upper bound of the pause grows as 200, 400, 800, 1600, 3200, 5000 ms, and the actual pause is a random number between zero and that bound.

src/fetch-with-retry.ts
import { setTimeout as delay } from "node:timers/promises";
import { UpstreamError } from "./errors.ts";

export type AttemptLog = {
  method: string;
  url: string;
  attempt: number;
  outcome: number | "timeout" | "network";
  ms: number;
  next: "done" | "retry" | "give-up";
  delayMs?: number;
};

export type RetryOptions = {
  retries?: number; // extra attempts after the first one
  baseDelayMs?: number;
  maxDelayMs?: number; // cap for the exponential delay
  attemptTimeoutMs?: number; // one attempt
  totalTimeoutMs?: number; // all attempts and pauses together
  maxRetryAfterMs?: number; // a longer Retry-After means "give up now"
  signal?: AbortSignal; // cancellation by the caller
  log?: (entry: AttemptLog) => void;
  sleep?: (ms: number, signal: AbortSignal) => Promise<unknown>;
  random?: () => number;
};

const RETRYABLE_STATUS = new Set([408, 429, 500, 502, 503, 504]);
const IDEMPOTENT_METHODS = new Set(["GET", "HEAD", "OPTIONS", "PUT", "DELETE"]);

// RFC 9110: do not repeat a non-idempotent request automatically,
// unless the API accepts an idempotency key for it
export function canRetry(method: string, headers: Headers): boolean {
  return IDEMPOTENT_METHODS.has(method) || headers.has("idempotency-key");
}

// Retry-After is either delay-seconds or an HTTP-date (RFC 9110, section 10.2.3)
export function parseRetryAfter(value: string | null, now = Date.now()): number | undefined {
  if (value === null) return undefined;
  const trimmed = value.trim();
  if (/^\d+$/.test(trimmed)) return Number(trimmed) * 1000;
  const date = Date.parse(trimmed);
  return Number.isNaN(date) ? undefined : Math.max(0, date - now);
}

// "Full Jitter": random_between(0, min(cap, base * 2 ** attempt))
export function backoffDelay(attempt: number, baseMs: number, capMs: number, random = Math.random): number {
  return Math.round(random() * Math.min(capMs, baseMs * 2 ** attempt));
}

export async function fetchWithRetry(
  url: string | URL,
  init: RequestInit = {},
  options: RetryOptions = {},
): Promise<Response> {
  const {
    retries = 3,
    baseDelayMs = 200,
    maxDelayMs = 5_000,
    attemptTimeoutMs = 2_000,
    totalTimeoutMs = 10_000,
    maxRetryAfterMs = 30_000,
    signal: callerSignal,
    log = () => {},
    sleep = (ms, signal) => delay(ms, undefined, { signal }),
    random = Math.random,
  } = options;

  const method = (init.method ?? "GET").toUpperCase();
  const maxAttempts = canRetry(method, new Headers(init.headers)) ? retries + 1 : 1;
  const deadline = AbortSignal.timeout(totalTimeoutMs);
  const deadlineAt = Date.now() + totalTimeoutMs;
  const overall = callerSignal ? AbortSignal.any([callerSignal, deadline]) : deadline;
  const target = String(url);

  for (let attempt = 1; ; attempt++) {
    const started = performance.now();
    const entry = (outcome: AttemptLog["outcome"]) => ({
      method, url: target, attempt, outcome, ms: Math.round(performance.now() - started),
    });
    let failure: UpstreamError;
    let outcome: AttemptLog["outcome"];

    try {
      const res = await fetch(url, {
        ...init,
        signal: AbortSignal.any([overall, AbortSignal.timeout(attemptTimeoutMs)]),
      });
      outcome = res.status;
      if (!RETRYABLE_STATUS.has(res.status)) {
        log({ ...entry(outcome), next: "done" });
        return res; // 2xx, 3xx and our own 4xx: the caller decides
      }
      await res.body?.cancel(); // release the connection
      failure = new UpstreamError(`${method} ${target} answered ${res.status}`, {
        kind: "http",
        status: res.status,
        attempts: attempt,
        retryAfterMs: parseRetryAfter(res.headers.get("retry-after")),
      });
    } catch (error) {
      if (callerSignal?.aborted) throw error; // cancelled by the caller: no retries
      const timedOut = error instanceof DOMException && error.name === "TimeoutError";
      outcome = timedOut ? "timeout" : "network";
      if (deadline.aborted) {
        log({ ...entry(outcome), next: "give-up" });
        throw new UpstreamError(`${method} ${target}: total timeout ${totalTimeoutMs} ms`, {
          kind: "deadline", attempts: attempt, cause: error,
        });
      }
      failure = new UpstreamError(`${method} ${target}: ${timedOut ? "attempt timed out" : "network error"}`, {
        kind: timedOut ? "timeout" : "network", attempts: attempt, cause: error,
      });
    }

    const wait = failure.retryAfterMs ?? backoffDelay(attempt - 1, baseDelayMs, maxDelayMs, random);
    const giveUp =
      attempt >= maxAttempts ||
      (failure.retryAfterMs ?? 0) > maxRetryAfterMs ||
      Date.now() + wait >= deadlineAt;
    if (giveUp) {
      log({ ...entry(outcome), next: "give-up" });
      throw failure;
    }
    log({ ...entry(outcome), next: "retry", delayMs: wait });
    try {
      await sleep(wait, overall);
    } catch (error) {
      if (callerSignal?.aborted) throw error;
      throw failure; // the deadline came during the pause
    }
  }
}
The fetch wrapper: timeouts, classification, Full Jitter, Retry-After, deadline and an attempt log
  • retries = 3 means up to four attempts in total. The retry count is always capped; an endless retry loop is a load test of someone else’s service at your expense.
  • res.body?.cancel() releases the connection of a response we are not going to read.
  • Before pausing, the wrapper checks whether the next attempt fits before the deadline; if not, it gives up at once instead of sleeping for nothing.
  • sleep and random are parameters: in tests the pauses are recorded instead of awaited, so the tests stay fast and deterministic.

Rate limits: 429, Retry-After and your own limiter

The 429 Too Many Requests status from RFC 6585 means the client has sent too many requests in a given amount of time. The server MAY include Retry-After — how long to wait before a new request. The standard does not define how the server counts requests or identifies the client: always check the limits in the documentation of the specific API. Per RFC 9110, Retry-After is either a number of seconds (Retry-After: 120) or an HTTP date (Retry-After: Fri, 31 Dec 1999 23:59:59 GMT); with a 503 it tells how long the service is expected to be unavailable.

  1. Retry-After is present

    Wait exactly as long as the server asks instead of our own exponent. parseRetryAfter understands both formats.

  2. Retry-After is too long

    If the wait exceeds maxRetryAfterMs (30 seconds by default) or does not fit the deadline, give up at once and pass the value in UpstreamError.retryAfterMs. A background job can reschedule itself for that time.

  3. Do not reach 429

    Limit your own pace in advance: if the provider allows 5 requests per second, the client should not send more on its own.

A token bucket works for your own limit: the bucket holds up to capacity tokens, they refill at ratePerSecond, and every request takes one. A short burst passes immediately, a long stream is smoothed to the given pace. AWS suggests the same mechanism for a retry budget: locally limit how many retries a client can afford at all.

src/rate-limiter.ts
import { setTimeout as delay } from "node:timers/promises";

// Token bucket: a burst of up to `capacity` requests, then `ratePerSecond` on average.
// Keeps us under the provider's limit instead of collecting 429 responses.
export class RateLimiter {
  #capacity: number;
  #ratePerSecond: number;
  #tokens: number;
  #updatedAt = performance.now();

  constructor({ capacity, ratePerSecond }: { capacity: number; ratePerSecond: number }) {
    this.#capacity = capacity;
    this.#ratePerSecond = ratePerSecond;
    this.#tokens = capacity;
  }

  async acquire(signal?: AbortSignal): Promise<void> {
    for (;;) {
      const now = performance.now();
      const refilled = ((now - this.#updatedAt) / 1000) * this.#ratePerSecond;
      this.#tokens = Math.min(this.#capacity, this.#tokens + refilled);
      this.#updatedAt = now;
      if (this.#tokens >= 1) {
        this.#tokens -= 1;
        return;
      }
      const waitMs = Math.ceil(((1 - this.#tokens) / this.#ratePerSecond) * 1000);
      await delay(waitMs, undefined, { signal });
    }
  }
}
A token bucket: a burst up to capacity, then ratePerSecond on average

Circuit breaker and graceful degradation

Retries are designed for short outages. If a service is down for half an hour, every request still runs through all attempts and timeouts, holding connections and users’ time. The Circuit Breaker pattern from the Azure Architecture Center solves this: after a series of failures the breaker “opens” and fails immediately without calling the service. The documentation stresses the difference: Retry expects the operation to eventually succeed, while Circuit Breaker prevents an operation that is likely to fail. They are combined by running retries through the breaker.

Closed

What happens
Calls go to the service, failures are counted
Transition
Failure threshold reached → Open

Open

What happens
A call fails at once with CircuitOpenError, the service is not touched
Transition
openMs elapsed → Half-Open

Half-Open

What happens
One trial call is let through
Transition
Success → Closed; failure → Open again
src/circuit-breaker.ts
import { CircuitOpenError, UpstreamError } from "./errors.ts";

export type CircuitState = "closed" | "open" | "half-open";

export type BreakerOptions = {
  name: string;
  failureThreshold?: number; // consecutive outages before opening
  openMs?: number; // how long to stay open before a trial call
  now?: () => number;
  onStateChange?: (from: CircuitState, to: CircuitState, name: string) => void;
};

export class CircuitBreaker {
  readonly name: string;
  #state: CircuitState = "closed";
  #failures = 0;
  #openedAt = 0;
  #trialInFlight = false;
  #threshold: number;
  #openMs: number;
  #now: () => number;
  #onStateChange: NonNullable<BreakerOptions["onStateChange"]>;

  constructor({ name, failureThreshold = 5, openMs = 30_000, now = Date.now, onStateChange = () => {} }: BreakerOptions) {
    this.name = name;
    this.#threshold = failureThreshold;
    this.#openMs = openMs;
    this.#now = now;
    this.#onStateChange = onStateChange;
  }

  get state(): CircuitState {
    return this.#state;
  }

  async run<T>(operation: () => Promise<T>): Promise<T> {
    if (this.#state === "open") {
      if (this.#now() - this.#openedAt < this.#openMs) throw new CircuitOpenError(this.name);
      this.#move("half-open");
    }
    if (this.#state === "half-open") {
      if (this.#trialInFlight) throw new CircuitOpenError(this.name); // one trial call at a time
      this.#trialInFlight = true;
    }
    try {
      const result = await operation();
      this.#failures = 0;
      if (this.#state === "half-open") this.#move("closed");
      return result;
    } catch (error) {
      // Only an outage of the external service counts; our own bugs and 4xx do not
      if (error instanceof UpstreamError) this.#recordFailure();
      throw error;
    } finally {
      this.#trialInFlight = false;
    }
  }

  #recordFailure() {
    this.#failures++;
    if (this.#state === "half-open" || this.#failures >= this.#threshold) {
      this.#failures = 0;
      this.#openedAt = this.#now();
      this.#move("open");
    }
  }

  #move(to: CircuitState) {
    const from = this.#state;
    this.#state = to;
    this.#onStateChange(from, to, this.name);
  }
}
The breaker: three states, one trial call, only external service failures count

Simplifications compared with Microsoft’s description: we count consecutive failures, while the documentation uses a time-based counter that resets periodically so that occasional failures do not open the circuit; in the half-open state we let one trial call through, while the documentation allows “a limited number”. For teaching code this is an honest trade-off; in production teams often use a ready-made library and tune thresholds from metrics.

Degradation is what the user gets while the service is unavailable. Microsoft’s documentation gives an example: when the breaker trips, the application returns default or cached responses and informs users that some data is temporarily unavailable. The exchange rate client below answers with the last good result if it is not older than maxStaleMs and marks it as stale-cache with its age — the UI can say “rate updated 12 minutes ago”.

src/rates-client.ts
import { CircuitBreaker } from "./circuit-breaker.ts";
import { CircuitOpenError, UpstreamError } from "./errors.ts";
import { fetchWithRetry, type RetryOptions } from "./fetch-with-retry.ts";
import { RateLimiter } from "./rate-limiter.ts";

export type Rates = { base: string; rates: Record<string, number> };
export type RatesResult = { data: Rates; source: "live" | "stale-cache"; ageMs: number };

type RatesClientOptions = {
  baseUrl: string;
  maxStaleMs?: number; // how old a cached answer may be during an outage
  retry?: RetryOptions;
  breaker?: CircuitBreaker;
  limiter?: RateLimiter;
  now?: () => number;
};

export function createRatesClient({
  baseUrl,
  maxStaleMs = 6 * 60 * 60_000,
  retry = {},
  breaker = new CircuitBreaker({ name: "rates-api" }),
  limiter = new RateLimiter({ capacity: 5, ratePerSecond: 5 }),
  now = Date.now,
}: RatesClientOptions) {
  let cache: { data: Rates; savedAt: number } | undefined;

  async function getRates(signal?: AbortSignal): Promise<RatesResult> {
    try {
      const data = await breaker.run(async () => {
        await limiter.acquire(signal);
        const res = await fetchWithRetry(new URL("/rates", baseUrl), { headers: { accept: "application/json" } }, { ...retry, signal });
        // A 4xx here is our bug or a wrong key: fail loudly, do not hide it behind the cache
        if (!res.ok) throw new Error(`Rates API rejected the request: ${res.status}`);
        return (await res.json()) as Rates;
      });
      cache = { data, savedAt: now() };
      return { data, source: "live", ageMs: 0 };
    } catch (error) {
      const outage = error instanceof UpstreamError || error instanceof CircuitOpenError;
      if (outage && cache && now() - cache.savedAt <= maxStaleMs) {
        return { data: cache.data, source: "stale-cache", ageMs: now() - cache.savedAt };
      }
      throw error;
    }
  }

  return { getRates, breaker };
}
The rates client: limiter, breaker, retries and a cached answer on failure

Testing: a capricious server, tests and a call log

Testing such a wrapper against a real API is awkward: outages cannot be reproduced on demand. Let’s start a local “capricious” server that answers from a script: a given status and headers, a hang or a dropped connection. The demo and the tests both use it.

src/flaky-server.ts
// A local "capricious" API for tests and demos: answers follow a script
import { createServer } from "node:http";
import type { AddressInfo } from "node:net";

export type Step =
  | { status: number; headers?: Record<string, string>; body?: unknown }
  | "hang" // never answers
  | "drop"; // closes the connection without a response

export async function startFlakyServer() {
  const scripts = new Map<string, Step[]>();
  const requests: { path: string; method: string; idempotencyKey: string | undefined }[] = [];

  const server = createServer((req, res) => {
    const path = new URL(req.url ?? "/", "http://localhost").pathname;
    const key = req.headers["idempotency-key"];
    requests.push({ path, method: req.method ?? "GET", idempotencyKey: typeof key === "string" ? key : undefined });
    const step = scripts.get(path)?.shift() ?? { status: 200, body: { ok: true } };
    if (step === "hang") return;
    if (step === "drop") return void req.socket.destroy();
    res.writeHead(step.status, { "content-type": "application/json", ...step.headers });
    res.end(JSON.stringify(step.body ?? {}));
  });

  await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
  const { port } = server.address() as AddressInfo;

  return {
    url: `http://127.0.0.1:${port}`,
    script(path: string, steps: Step[]) {
      scripts.set(path, [...steps]);
    },
    hits(path: string) {
      return requests.filter((r) => r.path === path);
    },
    async close() {
      server.closeAllConnections();
      await new Promise((resolve) => server.close(resolve));
    },
  };
}
A local capricious API: a script of responses per path and a request log
src/demo.ts
import { UpstreamError } from "./errors.ts";
import { fetchWithRetry } from "./fetch-with-retry.ts";
import { startFlakyServer } from "./flaky-server.ts";

const api = await startFlakyServer();
const log = (entry: object) => console.log(JSON.stringify(entry));

// GET: 503, a dropped connection, 429 with Retry-After, then success
api.script("/rates", [
  { status: 503 },
  "drop",
  { status: 429, headers: { "retry-after": "1" } },
  { status: 200, body: { base: "KZT", rates: { USD: 0.0019 } } },
]);
const res = await fetchWithRetry(`${api.url}/rates`, {}, { log });
console.log("result:", res.status, await res.json());

// POST without an idempotency key: no retries, the error goes to the caller
api.script("/orders", [{ status: 503 }]);
try {
  await fetchWithRetry(`${api.url}/orders`, { method: "POST", body: "{}" }, { log });
} catch (error) {
  if (!(error instanceof UpstreamError)) throw error;
  console.log("error:", error.kind, error.status, `attempts=${error.attempts}`);
}

await api.close();
Demo: GET survives a 503, a dropped connection and a 429; POST without a key is not retried
terminal
$ node src/demo.ts
{"method":"GET","url":"http://127.0.0.1:57524/rates","attempt":1,"outcome":503,"ms":12,"next":"retry","delayMs":163}
{"method":"GET","url":"http://127.0.0.1:57524/rates","attempt":2,"outcome":"network","ms":23,"next":"retry","delayMs":342}
{"method":"GET","url":"http://127.0.0.1:57524/rates","attempt":3,"outcome":429,"ms":2,"next":"retry","delayMs":1000}
{"method":"GET","url":"http://127.0.0.1:57524/rates","attempt":4,"outcome":200,"ms":16,"next":"done"}
result: 200 { base: 'KZT', rates: { USD: 0.0019 } }
{"method":"POST","url":"http://127.0.0.1:57524/orders","attempt":1,"outcome":503,"ms":4,"next":"give-up"}
error: http 503 attempts=1
Actual output on Node.js 24.18: pauses of 163 and 342 ms are Full Jitter, the 1000 ms pause comes from Retry-After; POST gave up after the first attempt

The log in the demo is the observability itself: one structured entry per attempt with method, URL, attempt number, outcome, duration and decision. Metrics are built from such entries: the ratio of attempts to calls, the distribution of outcomes (503, 429, timeout, network), latency percentiles of external calls and breaker state changes — the last one is worth an alert. Do not log the Authorization header, API keys or personal data from request bodies; structured logs and metrics are covered in the monitoring guide.

test/fetch-with-retry.test.ts
import assert from "node:assert/strict";
import { after, before, describe, it } from "node:test";
import { UpstreamError } from "../src/errors.ts";
import { backoffDelay, fetchWithRetry, parseRetryAfter, type RetryOptions } from "../src/fetch-with-retry.ts";
import { startFlakyServer } from "../src/flaky-server.ts";

let api: Awaited<ReturnType<typeof startFlakyServer>>;
before(async () => (api = await startFlakyServer()));
after(() => api.close());

// Pauses are recorded instead of waited for: tests stay fast and deterministic
function fast(extra: RetryOptions = {}) {
  const delays: number[] = [];
  const options: RetryOptions = {
    sleep: async (ms) => void delays.push(ms),
    random: () => 1, // upper bound of the jitter range
    ...extra,
  };
  return { delays, options };
}

describe("fetchWithRetry", () => {
  it("retries 5xx with exponential delays and returns the success", async () => {
    api.script("/a", [{ status: 503 }, { status: 502 }, { status: 200, body: { ok: 1 } }]);
    const { delays, options } = fast();
    const res = await fetchWithRetry(`${api.url}/a`, {}, options);
    assert.equal(res.status, 200);
    assert.deepEqual(await res.json(), { ok: 1 });
    assert.equal(api.hits("/a").length, 3);
    assert.deepEqual(delays, [200, 400]);
  });

  it("does not retry a 4xx and returns it to the caller", async () => {
    api.script("/b", [{ status: 400, body: { error: "bad" } }]);
    const res = await fetchWithRetry(`${api.url}/b`, {}, fast().options);
    assert.equal(res.status, 400);
    assert.equal(api.hits("/b").length, 1);
  });

  it("waits as long as Retry-After says on 429", async () => {
    api.script("/c", [{ status: 429, headers: { "retry-after": "3" } }, { status: 200 }]);
    const { delays, options } = fast();
    const res = await fetchWithRetry(`${api.url}/c`, {}, options);
    assert.equal(res.status, 200);
    assert.deepEqual(delays, [3000]);
  });

  it("gives up at once when Retry-After is longer than allowed", async () => {
    api.script("/d", [{ status: 503, headers: { "retry-after": "120" } }]);
    await assert.rejects(fetchWithRetry(`${api.url}/d`, {}, fast().options), (error: unknown) => {
      assert.ok(error instanceof UpstreamError);
      assert.equal(error.status, 503);
      assert.equal(error.retryAfterMs, 120_000); // the caller can reschedule the job
      return true;
    });
    assert.equal(api.hits("/d").length, 1);
  });

  it("aborts a hanging attempt by timeout and retries", async () => {
    api.script("/e", ["hang", { status: 200 }]);
    const entries: string[] = [];
    const { options } = fast({ attemptTimeoutMs: 100, log: (e) => entries.push(`${e.outcome}:${e.next}`) });
    const res = await fetchWithRetry(`${api.url}/e`, {}, options);
    assert.equal(res.status, 200);
    assert.deepEqual(entries, ["timeout:retry", "200:done"]);
  });

  it("retries a dropped connection", async () => {
    api.script("/f", ["drop", { status: 200 }]);
    const entries: string[] = [];
    const res = await fetchWithRetry(`${api.url}/f`, {}, fast({ log: (e) => entries.push(String(e.outcome)) }).options);
    assert.equal(res.status, 200);
    assert.deepEqual(entries, ["network", "200"]);
  });

  it("does not retry POST without an idempotency key", async () => {
    api.script("/g", [{ status: 503 }, { status: 201 }]);
    await assert.rejects(
      fetchWithRetry(`${api.url}/g`, { method: "POST", body: "{}" }, fast().options),
      (error: unknown) => error instanceof UpstreamError && error.attempts === 1,
    );
    assert.equal(api.hits("/g").length, 1);
  });

  it("retries POST with an idempotency key, sending the same key", async () => {
    api.script("/h", [{ status: 503 }, { status: 201 }]);
    const init = { method: "POST", body: "{}", headers: { "idempotency-key": "order-1042-payment" } };
    const res = await fetchWithRetry(`${api.url}/h`, init, fast().options);
    assert.equal(res.status, 201);
    assert.deepEqual(api.hits("/h").map((r) => r.idempotencyKey), ["order-1042-payment", "order-1042-payment"]);
  });

  it("stops at the total deadline even if attempts remain", async () => {
    api.script("/i", ["hang", "hang", "hang", "hang"]);
    const started = performance.now();
    await assert.rejects(
      fetchWithRetry(`${api.url}/i`, {}, { attemptTimeoutMs: 100, totalTimeoutMs: 250, baseDelayMs: 10 }),
      (error: unknown) => error instanceof UpstreamError && error.kind === "deadline",
    );
    assert.ok(performance.now() - started < 400);
  });

  it("does not retry when the caller cancels", async () => {
    api.script("/j", ["hang", { status: 200 }]);
    const controller = new AbortController();
    setTimeout(() => controller.abort(), 50);
    await assert.rejects(
      fetchWithRetry(`${api.url}/j`, {}, { signal: controller.signal }),
      (error: unknown) => error instanceof DOMException && error.name === "AbortError",
    );
    assert.equal(api.hits("/j").length, 1);
  });
});

describe("helpers", () => {
  it("parses Retry-After as seconds and as an HTTP date", () => {
    const now = Date.parse("2026-10-11T12:00:00Z");
    assert.equal(parseRetryAfter("120", now), 120_000);
    assert.equal(parseRetryAfter("Sun, 11 Oct 2026 12:00:30 GMT", now), 30_000);
    assert.equal(parseRetryAfter("soon", now), undefined);
    assert.equal(parseRetryAfter(null, now), undefined);
  });

  it("keeps full jitter within [0, min(cap, base * 2^n)]", () => {
    assert.equal(backoffDelay(0, 200, 5_000, () => 0), 0);
    assert.equal(backoffDelay(3, 200, 5_000, () => 1), 1_600);
    assert.equal(backoffDelay(10, 200, 5_000, () => 1), 5_000);
    assert.equal(backoffDelay(2, 200, 5_000, () => 0.5), 400);
  });
});
12 wrapper tests: 5xx, 4xx, 429 and Retry-After, hang, dropped connection, POST with and without a key, deadline, cancellation, parsing and jitter
test/resilience.test.ts
import assert from "node:assert/strict";
import { after, before, describe, it } from "node:test";
import { CircuitBreaker } from "../src/circuit-breaker.ts";
import { CircuitOpenError } from "../src/errors.ts";
import { startFlakyServer } from "../src/flaky-server.ts";
import { RateLimiter } from "../src/rate-limiter.ts";
import { createRatesClient } from "../src/rates-client.ts";

let api: Awaited<ReturnType<typeof startFlakyServer>>;
before(async () => (api = await startFlakyServer()));
after(() => api.close());

const noRetries = { retries: 0 };
const rates = { base: "KZT", rates: { USD: 0.0019 } };

describe("circuit breaker", () => {
  it("opens after N outages, fails fast, then closes after a successful trial", async () => {
    let clock = 0;
    const transitions: string[] = [];
    const breaker = new CircuitBreaker({
      name: "rates-api", failureThreshold: 3, openMs: 30_000, now: () => clock,
      onStateChange: (from, to) => transitions.push(`${from}->${to}`),
    });
    const client = createRatesClient({ baseUrl: api.url, breaker, retry: noRetries });
    api.script("/rates", [{ status: 503 }, { status: 503 }, { status: 503 }, { status: 200, body: rates }]);

    for (let i = 0; i < 3; i++) await assert.rejects(client.getRates());
    assert.equal(breaker.state, "open");
    await assert.rejects(client.getRates(), CircuitOpenError);
    assert.equal(api.hits("/rates").length, 3); // the 4th call never reached the server

    clock += 30_000; // the open period is over: one trial call goes through
    const result = await client.getRates();
    assert.equal(result.source, "live");
    assert.deepEqual(transitions, ["closed->open", "open->half-open", "half-open->closed"]);
  });

  it("does not count our own 4xx as an outage", async () => {
    const breaker = new CircuitBreaker({ name: "strict", failureThreshold: 1 });
    const client = createRatesClient({ baseUrl: api.url, breaker, retry: noRetries });
    api.script("/rates", [{ status: 401 }]);
    await assert.rejects(client.getRates(), /rejected the request: 401/);
    assert.equal(breaker.state, "closed");
  });
});

describe("degradation", () => {
  it("serves the last good answer from the cache during an outage", async () => {
    let clock = 1_000_000;
    const client = createRatesClient({ baseUrl: api.url, retry: noRetries, now: () => clock });
    api.script("/rates", [{ status: 200, body: rates }, { status: 503 }]);
    assert.equal((await client.getRates()).source, "live");

    clock += 60_000;
    const degraded = await client.getRates();
    assert.equal(degraded.source, "stale-cache");
    assert.equal(degraded.ageMs, 60_000);
    assert.deepEqual(degraded.data, rates);
  });

  it("fails when the cache is too old", async () => {
    let clock = 0;
    const client = createRatesClient({ baseUrl: api.url, retry: noRetries, maxStaleMs: 1_000, now: () => clock });
    api.script("/rates", [{ status: 200, body: rates }, { status: 503 }]);
    await client.getRates();
    clock += 5_000;
    await assert.rejects(client.getRates(), /answered 503/);
  });
});

describe("rate limiter", () => {
  it("lets a burst through and then spaces requests out", async () => {
    const limiter = new RateLimiter({ capacity: 2, ratePerSecond: 10 });
    const started = performance.now();
    await limiter.acquire();
    await limiter.acquire();
    assert.ok(performance.now() - started < 20, "the burst is immediate");
    await limiter.acquire();
    assert.ok(performance.now() - started >= 90, "the third request waits ~100 ms");
  });
});
5 tests: breaker opens and recovers, 4xx does not trip it, cached answer, too old cache, limiter
terminal

> resilient-fetch@1.0.0 test
> node --test

▶ fetchWithRetry
  ✔ retries 5xx with exponential delays and returns the success (17.6958ms)
  ✔ does not retry a 4xx and returns it to the caller (1.1994ms)
  ✔ waits as long as Retry-After says on 429 (14.2756ms)
  ✔ gives up at once when Retry-After is longer than allowed (13.248ms)
  ✔ aborts a hanging attempt by timeout and retries (120.7663ms)
  ✔ retries a dropped connection (18.6495ms)
  ✔ does not retry POST without an idempotency key (3.393ms)
  ✔ retries POST with an idempotency key, sending the same key (12.1761ms)
  ✔ stops at the total deadline even if attempts remain (262.2019ms)
  ✔ does not retry when the caller cancels (63.5269ms)
✔ fetchWithRetry (532.1418ms)
▶ helpers
  ✔ parses Retry-After as seconds and as an HTTP date (0.2605ms)
  ✔ keeps full jitter within [0, min(cap, base * 2^n)] (0.1896ms)
✔ helpers (0.6801ms)
▶ circuit breaker
  ✔ opens after N outages, fails fast, then closes after a successful trial (39.2041ms)
  ✔ does not count our own 4xx as an outage (2.3658ms)
✔ circuit breaker (45.7581ms)
▶ degradation
  ✔ serves the last good answer from the cache during an outage (10.3849ms)
  ✔ fails when the cache is too old (31.4225ms)
✔ degradation (42.0438ms)
▶ rate limiter
  ✔ lets a burst through and then spaces requests out (108.3147ms)
✔ rate limiter (108.5081ms)
ℹ tests 17
ℹ suites 5
ℹ pass 17
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 670.8636
Actual npm test output on Node.js 24.18
terminal
$ npm run typecheck

> resilient-fetch@1.0.0 typecheck
> tsc
TypeScript 7.0.2 type check passed without errors

The tests were checked in reverse: remove the idempotency condition and the test about POST without a key fails; ignore Retry-After and the 429 test fails. The hang test waits for a real 100 ms timeout and the deadline test for 250 ms; the other pauses are replaced with recording into an array.

  • fetch without a timeout: a hung service holds requests and connections
  • Retrying everything, including 400 and 401
  • Automatically retrying POST without an idempotency key — double payments and orders
  • Retries without a pause or with the same pause for all clients, no jitter
  • Retries at every layer: HTTP client, service, queue — the load multiplies
  • Retry-After ignored and the provider limit not respected in advance
  • The breaker trips on our own 4xx responses
  • Stale cached data shown where accuracy matters: payments, stock levels
How this was verified3 · passedchecked 11 October 2026

“Run” means the code was executed in a scratch project or container and the output in the article comes from that run. “Partial” means it was checked against stubs instead of external services. “Not run” means it needs a device, an account or a key; the steps were checked against the official documentation. The verification project files and run logs are kept in the InterLAB archive.

Tell us about your task — we'll review your request and get back to you

Got a project?
We'll scope and estimate it — for free.

Tell us about the product or the problem. After reviewing the task we'll come back with an architecture, a timeline and a price for the first version.

  • Scope and risks estimated before we start
  • NDA — yours or ours
  • Work starts within a week

Please do not send passwords, access keys or source code through this form — we will agree a safe way to share access separately.

By clicking the button you agree to the processing of your personal data.