Every call to a third-party API needs a per-attempt timeout and an overall deadline; retry only transient failures (network, timeout, 408, 429, 5xx) and only for idempotent requests, with exponential pauses, random jitter and respect for the Retry-After header. If the service stays down, a circuit breaker stops calling it and the application answers from a cache or with an honest fallback instead of failing its own pages.
After this you will be able to
- set a per-attempt timeout and an overall deadline with
AbortSignal.timeoutandAbortSignal.anyand tell a timeout from a cancellation - classify external API errors and retry only transient failures of idempotent requests, and
POSTonly with anIdempotency-Key - implement exponential backoff with Full Jitter, respect
429andRetry-Afterand limit your own pace with a token bucket - add a circuit breaker and a cached fallback and test everything against a local capricious server
Know beforehand
Verified with
Node.js 24 LTS (24.18)TypeScript 7.0 (tsc 7.0.2)@types/node 24Fetch API, AbortSignal.timeout / AbortSignal.anynode:test
Checked against docs: October 11, 2026
Contents
A third-party API can always fail
A payment gateway, exchange rates, a CRM, a delivery service — all of them are someone else’s servers with their own outages, deployments and limits. If code simply calls await fetch(url), a hung service holds your request and connection, a burst of 503s turns into errors on your pages, and naive instant retries finish off a service that is trying to recover. The task is to decide up front, for every call, how long to wait, what to retry and what to show when there is no answer.
No connection, connection reset
- What it looks like in Node.js
fetchrejects with aTypeError- What we do
- Retry after a pause if the request is idempotent
Service hangs
- What it looks like in Node.js
- No response; with a timeout —
TimeoutError - What we do
- Per-attempt timeout, then retry
Transient server error
- What it looks like in Node.js
500,502,503,504,408- What we do
- Retry with an exponential pause
Rate limit exceeded
- What it looks like in Node.js
429, often withRetry-After- What we do
- Wait as long as the server asks or hand the job to the background
Error in our request
- What it looks like in Node.js
400,401,403,404,422- What we do
- Do not retry: the answer will not change
Service down for a long time
- What it looks like in Node.js
- Nothing but failures in a row
- What we do
- Circuit breaker and degradation: cache or fallback
- 01Application serviceCache and degradation: what to show when there is no answer
- 02Circuit breakerDo not call a service that is down
- 03Rate limiterStay under the provider limit
- 04fetchWithRetryTimeouts, error classification, backoff, Retry-After
- 05External APISomeone else’s server with its own outages and limits
This guide is for backend developers who are comfortable with async/await and error handling. The code is tested on Node.js 24.18 and TypeScript 7.0; there are no runtime dependencies, only the built-in fetch, AbortSignal and node:test. Node.js 24 runs .ts files directly: type stripping is stable since 24.12, and tsc checks the types. This guide covers the mechanics of a reliable call; a typed client for a specific API with response schemas is a separate recipe.
mkdir resilient-fetch
cd resilient-fetch
npm init -y
npm pkg set type=module
npm pkg set private=true --json
npm pkg delete main scripts.test
npm pkg set scripts.test="node --test" scripts.typecheck="tsc"
npm install -D typescript @types/node@24
mkdir src test{
"compilerOptions": {
"target": "es2024",
"module": "nodenext",
"strict": true,
"noEmit": true,
"allowImportingTsExtensions": true,
"erasableSyntaxOnly": true,
"verbatimModuleSyntax": true,
"skipLibCheck": true,
"types": ["node"]
},
"include": ["src", "test"]
}Timeouts: per attempt and for the whole operation
You need two timeouts. The attempt timeout cuts off one hung request and gives a retry a chance. The overall deadline limits the whole operation including retries and pauses: the user or the calling service will not wait longer than that, however many attempts remain. Both are standard signals: AbortSignal.timeout(ms) fires on its own after the given time, and AbortSignal.any([...]) combines it with a cancellation signal from the caller. According to MDN, a timeout aborts with a TimeoutError while a manual abort gives an AbortError — this is how the wrapper tells “the server was too slow” from “we were cancelled”.
How to pick the number. The AWS Builders Library article on timeouts advises starting from the latency metrics of the downstream service: choose an acceptable rate of false timeouts, for example 0.1%, and take the matching percentile — p99.9. If there are no metrics yet, in our experience 2–5 seconds per attempt is a reasonable start for a typical JSON API, as long as you record the duration of every call and tune the value on real data.
In the wrapper below every attempt gets AbortSignal.any([overall, AbortSignal.timeout(attemptTimeoutMs)]), where overall is the deadline plus the caller’s cancellation. The pause between attempts listens to overall too, so the deadline does not “oversleep” a long pause.
Which errors to retry and which not
A retry only helps against transient failures. A 400 or 401 for the same request will not change no matter how often you repeat it — that is an error in the data or the key, and it must be surfaced and fixed. The second question is whether a retry is safe at all. RFC 9110 defines idempotent methods — PUT, DELETE and the safe GET, HEAD, OPTIONS: several identical requests have the same effect as one, which is exactly why they can be repeated after a communication failure. For non-idempotent methods the standard says a client SHOULD NOT retry them automatically unless it knows the request is actually idempotent. AWS puts it shorter: APIs with side effects are not safe to retry unless they provide idempotency.
Network error, connection reset
- Retry?
- Yes, if the method is idempotent or there is a key
- Comment
- The request may have reached the server before the reset
Attempt timeout
- Retry?
- Yes, under the same condition
- Comment
- The server may have performed the action
408 Request Timeout
- Retry?
- Yes
- Comment
- RFC 9110 allows the client to repeat the request
429 Too Many Requests
- Retry?
- Yes, after
Retry-After - Comment
- If the wait is longer than allowed, hand off to the background
500, 502, 503, 504
- Retry?
- Yes, with an exponential pause
- Comment
- A
503may also come withRetry-After
Other 4xx
- Retry?
- No
- Comment
- The response is returned to the caller as is
Cancelled by the caller
- Retry?
- Never
- Comment
- The user has left — nobody needs the answer
For POST the wrapper retries only when the headers contain Idempotency-Key: a server that supports such a key recognises the repeat and returns the original result instead of creating a second record. The server side of this is covered in the API design guide; the key is generated once per operation and is the same in every attempt.
A failure of the external service after all attempts is expressed by a dedicated error class. This matters for the next layers: the circuit breaker and the cache react only to UpstreamError, while a 4xx about our own request must neither trip the breaker nor hide behind a stale cache.
export type FailureKind = "network" | "timeout" | "deadline" | "http";
// The external service failed: network, timeout or 408/429/5xx after all attempts.
// A 4xx about our own request is not an UpstreamError: the response is returned as is.
export class UpstreamError extends Error {
readonly kind: FailureKind;
readonly status: number | undefined;
readonly attempts: number;
readonly retryAfterMs: number | undefined;
constructor(
message: string,
details: { kind: FailureKind; attempts: number; status?: number; retryAfterMs?: number; cause?: unknown },
) {
super(message, { cause: details.cause });
this.name = "UpstreamError";
this.kind = details.kind;
this.status = details.status;
this.attempts = details.attempts;
this.retryAfterMs = details.retryAfterMs;
}
}
export class CircuitOpenError extends Error {
constructor(name: string) {
super(`Circuit "${name}" is open: the call was not made`);
this.name = "CircuitOpenError";
}
}Exponential backoff and jitter
Retrying immediately is pointless: a service that has just answered 503 will most likely answer the same a millisecond later. Exponential backoff doubles the pause after every failure, and a cap keeps it from growing forever. But if an outage hits thousands of clients at once, their retries come back in synchronised waves. The AWS Architecture Blog post “Exponential Backoff And Jitter” (Marc Brooker) shows this with a simulation and compares variants of random spread — jitter:
Capped exponential
- Formula from the AWS post
sleep = min(cap, base * 2 ** attempt)- What the post says
- Clients keep contending in waves
Full Jitter
- Formula from the AWS post
sleep = random_between(0, min(cap, base * 2 ** attempt))- What the post says
- Least work for the server, slightly more time
Equal Jitter
- Formula from the AWS post
temp = min(cap, base * 2 ** attempt);sleep = temp / 2 + random_between(0, temp / 2)- What the post says
- The worst of the jittered variants
Decorrelated Jitter
- Formula from the AWS post
sleep = min(cap, random_between(base, sleep * 3))- What the post says
- The choice between it and Full Jitter is, in the author’s words, “less clear”
The post concludes that jitter should be considered a standard approach for remote clients. We use Full Jitter — it is the simplest and causes the fewest extra calls. With base = 200 ms and cap = 5000 ms the upper bound of the pause grows as 200, 400, 800, 1600, 3200, 5000 ms, and the actual pause is a random number between zero and that bound.
import { setTimeout as delay } from "node:timers/promises";
import { UpstreamError } from "./errors.ts";
export type AttemptLog = {
method: string;
url: string;
attempt: number;
outcome: number | "timeout" | "network";
ms: number;
next: "done" | "retry" | "give-up";
delayMs?: number;
};
export type RetryOptions = {
retries?: number; // extra attempts after the first one
baseDelayMs?: number;
maxDelayMs?: number; // cap for the exponential delay
attemptTimeoutMs?: number; // one attempt
totalTimeoutMs?: number; // all attempts and pauses together
maxRetryAfterMs?: number; // a longer Retry-After means "give up now"
signal?: AbortSignal; // cancellation by the caller
log?: (entry: AttemptLog) => void;
sleep?: (ms: number, signal: AbortSignal) => Promise<unknown>;
random?: () => number;
};
const RETRYABLE_STATUS = new Set([408, 429, 500, 502, 503, 504]);
const IDEMPOTENT_METHODS = new Set(["GET", "HEAD", "OPTIONS", "PUT", "DELETE"]);
// RFC 9110: do not repeat a non-idempotent request automatically,
// unless the API accepts an idempotency key for it
export function canRetry(method: string, headers: Headers): boolean {
return IDEMPOTENT_METHODS.has(method) || headers.has("idempotency-key");
}
// Retry-After is either delay-seconds or an HTTP-date (RFC 9110, section 10.2.3)
export function parseRetryAfter(value: string | null, now = Date.now()): number | undefined {
if (value === null) return undefined;
const trimmed = value.trim();
if (/^\d+$/.test(trimmed)) return Number(trimmed) * 1000;
const date = Date.parse(trimmed);
return Number.isNaN(date) ? undefined : Math.max(0, date - now);
}
// "Full Jitter": random_between(0, min(cap, base * 2 ** attempt))
export function backoffDelay(attempt: number, baseMs: number, capMs: number, random = Math.random): number {
return Math.round(random() * Math.min(capMs, baseMs * 2 ** attempt));
}
export async function fetchWithRetry(
url: string | URL,
init: RequestInit = {},
options: RetryOptions = {},
): Promise<Response> {
const {
retries = 3,
baseDelayMs = 200,
maxDelayMs = 5_000,
attemptTimeoutMs = 2_000,
totalTimeoutMs = 10_000,
maxRetryAfterMs = 30_000,
signal: callerSignal,
log = () => {},
sleep = (ms, signal) => delay(ms, undefined, { signal }),
random = Math.random,
} = options;
const method = (init.method ?? "GET").toUpperCase();
const maxAttempts = canRetry(method, new Headers(init.headers)) ? retries + 1 : 1;
const deadline = AbortSignal.timeout(totalTimeoutMs);
const deadlineAt = Date.now() + totalTimeoutMs;
const overall = callerSignal ? AbortSignal.any([callerSignal, deadline]) : deadline;
const target = String(url);
for (let attempt = 1; ; attempt++) {
const started = performance.now();
const entry = (outcome: AttemptLog["outcome"]) => ({
method, url: target, attempt, outcome, ms: Math.round(performance.now() - started),
});
let failure: UpstreamError;
let outcome: AttemptLog["outcome"];
try {
const res = await fetch(url, {
...init,
signal: AbortSignal.any([overall, AbortSignal.timeout(attemptTimeoutMs)]),
});
outcome = res.status;
if (!RETRYABLE_STATUS.has(res.status)) {
log({ ...entry(outcome), next: "done" });
return res; // 2xx, 3xx and our own 4xx: the caller decides
}
await res.body?.cancel(); // release the connection
failure = new UpstreamError(`${method} ${target} answered ${res.status}`, {
kind: "http",
status: res.status,
attempts: attempt,
retryAfterMs: parseRetryAfter(res.headers.get("retry-after")),
});
} catch (error) {
if (callerSignal?.aborted) throw error; // cancelled by the caller: no retries
const timedOut = error instanceof DOMException && error.name === "TimeoutError";
outcome = timedOut ? "timeout" : "network";
if (deadline.aborted) {
log({ ...entry(outcome), next: "give-up" });
throw new UpstreamError(`${method} ${target}: total timeout ${totalTimeoutMs} ms`, {
kind: "deadline", attempts: attempt, cause: error,
});
}
failure = new UpstreamError(`${method} ${target}: ${timedOut ? "attempt timed out" : "network error"}`, {
kind: timedOut ? "timeout" : "network", attempts: attempt, cause: error,
});
}
const wait = failure.retryAfterMs ?? backoffDelay(attempt - 1, baseDelayMs, maxDelayMs, random);
const giveUp =
attempt >= maxAttempts ||
(failure.retryAfterMs ?? 0) > maxRetryAfterMs ||
Date.now() + wait >= deadlineAt;
if (giveUp) {
log({ ...entry(outcome), next: "give-up" });
throw failure;
}
log({ ...entry(outcome), next: "retry", delayMs: wait });
try {
await sleep(wait, overall);
} catch (error) {
if (callerSignal?.aborted) throw error;
throw failure; // the deadline came during the pause
}
}
}retries = 3means up to four attempts in total. The retry count is always capped; an endless retry loop is a load test of someone else’s service at your expense.res.body?.cancel()releases the connection of a response we are not going to read.- Before pausing, the wrapper checks whether the next attempt fits before the deadline; if not, it gives up at once instead of sleeping for nothing.
sleepandrandomare parameters: in tests the pauses are recorded instead of awaited, so the tests stay fast and deterministic.
Rate limits: 429, Retry-After and your own limiter
The 429 Too Many Requests status from RFC 6585 means the client has sent too many requests in a given amount of time. The server MAY include Retry-After — how long to wait before a new request. The standard does not define how the server counts requests or identifies the client: always check the limits in the documentation of the specific API. Per RFC 9110, Retry-After is either a number of seconds (Retry-After: 120) or an HTTP date (Retry-After: Fri, 31 Dec 1999 23:59:59 GMT); with a 503 it tells how long the service is expected to be unavailable.
Retry-After is present
Wait exactly as long as the server asks instead of our own exponent.
parseRetryAfterunderstands both formats.Retry-After is too long
If the wait exceeds
maxRetryAfterMs(30 seconds by default) or does not fit the deadline, give up at once and pass the value inUpstreamError.retryAfterMs. A background job can reschedule itself for that time.Do not reach 429
Limit your own pace in advance: if the provider allows 5 requests per second, the client should not send more on its own.
A token bucket works for your own limit: the bucket holds up to capacity tokens, they refill at ratePerSecond, and every request takes one. A short burst passes immediately, a long stream is smoothed to the given pace. AWS suggests the same mechanism for a retry budget: locally limit how many retries a client can afford at all.
import { setTimeout as delay } from "node:timers/promises";
// Token bucket: a burst of up to `capacity` requests, then `ratePerSecond` on average.
// Keeps us under the provider's limit instead of collecting 429 responses.
export class RateLimiter {
#capacity: number;
#ratePerSecond: number;
#tokens: number;
#updatedAt = performance.now();
constructor({ capacity, ratePerSecond }: { capacity: number; ratePerSecond: number }) {
this.#capacity = capacity;
this.#ratePerSecond = ratePerSecond;
this.#tokens = capacity;
}
async acquire(signal?: AbortSignal): Promise<void> {
for (;;) {
const now = performance.now();
const refilled = ((now - this.#updatedAt) / 1000) * this.#ratePerSecond;
this.#tokens = Math.min(this.#capacity, this.#tokens + refilled);
this.#updatedAt = now;
if (this.#tokens >= 1) {
this.#tokens -= 1;
return;
}
const waitMs = Math.ceil(((1 - this.#tokens) / this.#ratePerSecond) * 1000);
await delay(waitMs, undefined, { signal });
}
}
}Circuit breaker and graceful degradation
Retries are designed for short outages. If a service is down for half an hour, every request still runs through all attempts and timeouts, holding connections and users’ time. The Circuit Breaker pattern from the Azure Architecture Center solves this: after a series of failures the breaker “opens” and fails immediately without calling the service. The documentation stresses the difference: Retry expects the operation to eventually succeed, while Circuit Breaker prevents an operation that is likely to fail. They are combined by running retries through the breaker.
Closed
- What happens
- Calls go to the service, failures are counted
- Transition
- Failure threshold reached → Open
Open
- What happens
- A call fails at once with
CircuitOpenError, the service is not touched - Transition
openMselapsed → Half-Open
Half-Open
- What happens
- One trial call is let through
- Transition
- Success → Closed; failure → Open again
import { CircuitOpenError, UpstreamError } from "./errors.ts";
export type CircuitState = "closed" | "open" | "half-open";
export type BreakerOptions = {
name: string;
failureThreshold?: number; // consecutive outages before opening
openMs?: number; // how long to stay open before a trial call
now?: () => number;
onStateChange?: (from: CircuitState, to: CircuitState, name: string) => void;
};
export class CircuitBreaker {
readonly name: string;
#state: CircuitState = "closed";
#failures = 0;
#openedAt = 0;
#trialInFlight = false;
#threshold: number;
#openMs: number;
#now: () => number;
#onStateChange: NonNullable<BreakerOptions["onStateChange"]>;
constructor({ name, failureThreshold = 5, openMs = 30_000, now = Date.now, onStateChange = () => {} }: BreakerOptions) {
this.name = name;
this.#threshold = failureThreshold;
this.#openMs = openMs;
this.#now = now;
this.#onStateChange = onStateChange;
}
get state(): CircuitState {
return this.#state;
}
async run<T>(operation: () => Promise<T>): Promise<T> {
if (this.#state === "open") {
if (this.#now() - this.#openedAt < this.#openMs) throw new CircuitOpenError(this.name);
this.#move("half-open");
}
if (this.#state === "half-open") {
if (this.#trialInFlight) throw new CircuitOpenError(this.name); // one trial call at a time
this.#trialInFlight = true;
}
try {
const result = await operation();
this.#failures = 0;
if (this.#state === "half-open") this.#move("closed");
return result;
} catch (error) {
// Only an outage of the external service counts; our own bugs and 4xx do not
if (error instanceof UpstreamError) this.#recordFailure();
throw error;
} finally {
this.#trialInFlight = false;
}
}
#recordFailure() {
this.#failures++;
if (this.#state === "half-open" || this.#failures >= this.#threshold) {
this.#failures = 0;
this.#openedAt = this.#now();
this.#move("open");
}
}
#move(to: CircuitState) {
const from = this.#state;
this.#state = to;
this.#onStateChange(from, to, this.name);
}
}Simplifications compared with Microsoft’s description: we count consecutive failures, while the documentation uses a time-based counter that resets periodically so that occasional failures do not open the circuit; in the half-open state we let one trial call through, while the documentation allows “a limited number”. For teaching code this is an honest trade-off; in production teams often use a ready-made library and tune thresholds from metrics.
Degradation is what the user gets while the service is unavailable. Microsoft’s documentation gives an example: when the breaker trips, the application returns default or cached responses and informs users that some data is temporarily unavailable. The exchange rate client below answers with the last good result if it is not older than maxStaleMs and marks it as stale-cache with its age — the UI can say “rate updated 12 minutes ago”.
import { CircuitBreaker } from "./circuit-breaker.ts";
import { CircuitOpenError, UpstreamError } from "./errors.ts";
import { fetchWithRetry, type RetryOptions } from "./fetch-with-retry.ts";
import { RateLimiter } from "./rate-limiter.ts";
export type Rates = { base: string; rates: Record<string, number> };
export type RatesResult = { data: Rates; source: "live" | "stale-cache"; ageMs: number };
type RatesClientOptions = {
baseUrl: string;
maxStaleMs?: number; // how old a cached answer may be during an outage
retry?: RetryOptions;
breaker?: CircuitBreaker;
limiter?: RateLimiter;
now?: () => number;
};
export function createRatesClient({
baseUrl,
maxStaleMs = 6 * 60 * 60_000,
retry = {},
breaker = new CircuitBreaker({ name: "rates-api" }),
limiter = new RateLimiter({ capacity: 5, ratePerSecond: 5 }),
now = Date.now,
}: RatesClientOptions) {
let cache: { data: Rates; savedAt: number } | undefined;
async function getRates(signal?: AbortSignal): Promise<RatesResult> {
try {
const data = await breaker.run(async () => {
await limiter.acquire(signal);
const res = await fetchWithRetry(new URL("/rates", baseUrl), { headers: { accept: "application/json" } }, { ...retry, signal });
// A 4xx here is our bug or a wrong key: fail loudly, do not hide it behind the cache
if (!res.ok) throw new Error(`Rates API rejected the request: ${res.status}`);
return (await res.json()) as Rates;
});
cache = { data, savedAt: now() };
return { data, source: "live", ageMs: 0 };
} catch (error) {
const outage = error instanceof UpstreamError || error instanceof CircuitOpenError;
if (outage && cache && now() - cache.savedAt <= maxStaleMs) {
return { data: cache.data, source: "stale-cache", ageMs: now() - cache.savedAt };
}
throw error;
}
}
return { getRates, breaker };
}Testing: a capricious server, tests and a call log
Testing such a wrapper against a real API is awkward: outages cannot be reproduced on demand. Let’s start a local “capricious” server that answers from a script: a given status and headers, a hang or a dropped connection. The demo and the tests both use it.
// A local "capricious" API for tests and demos: answers follow a script
import { createServer } from "node:http";
import type { AddressInfo } from "node:net";
export type Step =
| { status: number; headers?: Record<string, string>; body?: unknown }
| "hang" // never answers
| "drop"; // closes the connection without a response
export async function startFlakyServer() {
const scripts = new Map<string, Step[]>();
const requests: { path: string; method: string; idempotencyKey: string | undefined }[] = [];
const server = createServer((req, res) => {
const path = new URL(req.url ?? "/", "http://localhost").pathname;
const key = req.headers["idempotency-key"];
requests.push({ path, method: req.method ?? "GET", idempotencyKey: typeof key === "string" ? key : undefined });
const step = scripts.get(path)?.shift() ?? { status: 200, body: { ok: true } };
if (step === "hang") return;
if (step === "drop") return void req.socket.destroy();
res.writeHead(step.status, { "content-type": "application/json", ...step.headers });
res.end(JSON.stringify(step.body ?? {}));
});
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
const { port } = server.address() as AddressInfo;
return {
url: `http://127.0.0.1:${port}`,
script(path: string, steps: Step[]) {
scripts.set(path, [...steps]);
},
hits(path: string) {
return requests.filter((r) => r.path === path);
},
async close() {
server.closeAllConnections();
await new Promise((resolve) => server.close(resolve));
},
};
}import { UpstreamError } from "./errors.ts";
import { fetchWithRetry } from "./fetch-with-retry.ts";
import { startFlakyServer } from "./flaky-server.ts";
const api = await startFlakyServer();
const log = (entry: object) => console.log(JSON.stringify(entry));
// GET: 503, a dropped connection, 429 with Retry-After, then success
api.script("/rates", [
{ status: 503 },
"drop",
{ status: 429, headers: { "retry-after": "1" } },
{ status: 200, body: { base: "KZT", rates: { USD: 0.0019 } } },
]);
const res = await fetchWithRetry(`${api.url}/rates`, {}, { log });
console.log("result:", res.status, await res.json());
// POST without an idempotency key: no retries, the error goes to the caller
api.script("/orders", [{ status: 503 }]);
try {
await fetchWithRetry(`${api.url}/orders`, { method: "POST", body: "{}" }, { log });
} catch (error) {
if (!(error instanceof UpstreamError)) throw error;
console.log("error:", error.kind, error.status, `attempts=${error.attempts}`);
}
await api.close();$ node src/demo.ts
{"method":"GET","url":"http://127.0.0.1:57524/rates","attempt":1,"outcome":503,"ms":12,"next":"retry","delayMs":163}
{"method":"GET","url":"http://127.0.0.1:57524/rates","attempt":2,"outcome":"network","ms":23,"next":"retry","delayMs":342}
{"method":"GET","url":"http://127.0.0.1:57524/rates","attempt":3,"outcome":429,"ms":2,"next":"retry","delayMs":1000}
{"method":"GET","url":"http://127.0.0.1:57524/rates","attempt":4,"outcome":200,"ms":16,"next":"done"}
result: 200 { base: 'KZT', rates: { USD: 0.0019 } }
{"method":"POST","url":"http://127.0.0.1:57524/orders","attempt":1,"outcome":503,"ms":4,"next":"give-up"}
error: http 503 attempts=1The log in the demo is the observability itself: one structured entry per attempt with method, URL, attempt number, outcome, duration and decision. Metrics are built from such entries: the ratio of attempts to calls, the distribution of outcomes (503, 429, timeout, network), latency percentiles of external calls and breaker state changes — the last one is worth an alert. Do not log the Authorization header, API keys or personal data from request bodies; structured logs and metrics are covered in the monitoring guide.
import assert from "node:assert/strict";
import { after, before, describe, it } from "node:test";
import { UpstreamError } from "../src/errors.ts";
import { backoffDelay, fetchWithRetry, parseRetryAfter, type RetryOptions } from "../src/fetch-with-retry.ts";
import { startFlakyServer } from "../src/flaky-server.ts";
let api: Awaited<ReturnType<typeof startFlakyServer>>;
before(async () => (api = await startFlakyServer()));
after(() => api.close());
// Pauses are recorded instead of waited for: tests stay fast and deterministic
function fast(extra: RetryOptions = {}) {
const delays: number[] = [];
const options: RetryOptions = {
sleep: async (ms) => void delays.push(ms),
random: () => 1, // upper bound of the jitter range
...extra,
};
return { delays, options };
}
describe("fetchWithRetry", () => {
it("retries 5xx with exponential delays and returns the success", async () => {
api.script("/a", [{ status: 503 }, { status: 502 }, { status: 200, body: { ok: 1 } }]);
const { delays, options } = fast();
const res = await fetchWithRetry(`${api.url}/a`, {}, options);
assert.equal(res.status, 200);
assert.deepEqual(await res.json(), { ok: 1 });
assert.equal(api.hits("/a").length, 3);
assert.deepEqual(delays, [200, 400]);
});
it("does not retry a 4xx and returns it to the caller", async () => {
api.script("/b", [{ status: 400, body: { error: "bad" } }]);
const res = await fetchWithRetry(`${api.url}/b`, {}, fast().options);
assert.equal(res.status, 400);
assert.equal(api.hits("/b").length, 1);
});
it("waits as long as Retry-After says on 429", async () => {
api.script("/c", [{ status: 429, headers: { "retry-after": "3" } }, { status: 200 }]);
const { delays, options } = fast();
const res = await fetchWithRetry(`${api.url}/c`, {}, options);
assert.equal(res.status, 200);
assert.deepEqual(delays, [3000]);
});
it("gives up at once when Retry-After is longer than allowed", async () => {
api.script("/d", [{ status: 503, headers: { "retry-after": "120" } }]);
await assert.rejects(fetchWithRetry(`${api.url}/d`, {}, fast().options), (error: unknown) => {
assert.ok(error instanceof UpstreamError);
assert.equal(error.status, 503);
assert.equal(error.retryAfterMs, 120_000); // the caller can reschedule the job
return true;
});
assert.equal(api.hits("/d").length, 1);
});
it("aborts a hanging attempt by timeout and retries", async () => {
api.script("/e", ["hang", { status: 200 }]);
const entries: string[] = [];
const { options } = fast({ attemptTimeoutMs: 100, log: (e) => entries.push(`${e.outcome}:${e.next}`) });
const res = await fetchWithRetry(`${api.url}/e`, {}, options);
assert.equal(res.status, 200);
assert.deepEqual(entries, ["timeout:retry", "200:done"]);
});
it("retries a dropped connection", async () => {
api.script("/f", ["drop", { status: 200 }]);
const entries: string[] = [];
const res = await fetchWithRetry(`${api.url}/f`, {}, fast({ log: (e) => entries.push(String(e.outcome)) }).options);
assert.equal(res.status, 200);
assert.deepEqual(entries, ["network", "200"]);
});
it("does not retry POST without an idempotency key", async () => {
api.script("/g", [{ status: 503 }, { status: 201 }]);
await assert.rejects(
fetchWithRetry(`${api.url}/g`, { method: "POST", body: "{}" }, fast().options),
(error: unknown) => error instanceof UpstreamError && error.attempts === 1,
);
assert.equal(api.hits("/g").length, 1);
});
it("retries POST with an idempotency key, sending the same key", async () => {
api.script("/h", [{ status: 503 }, { status: 201 }]);
const init = { method: "POST", body: "{}", headers: { "idempotency-key": "order-1042-payment" } };
const res = await fetchWithRetry(`${api.url}/h`, init, fast().options);
assert.equal(res.status, 201);
assert.deepEqual(api.hits("/h").map((r) => r.idempotencyKey), ["order-1042-payment", "order-1042-payment"]);
});
it("stops at the total deadline even if attempts remain", async () => {
api.script("/i", ["hang", "hang", "hang", "hang"]);
const started = performance.now();
await assert.rejects(
fetchWithRetry(`${api.url}/i`, {}, { attemptTimeoutMs: 100, totalTimeoutMs: 250, baseDelayMs: 10 }),
(error: unknown) => error instanceof UpstreamError && error.kind === "deadline",
);
assert.ok(performance.now() - started < 400);
});
it("does not retry when the caller cancels", async () => {
api.script("/j", ["hang", { status: 200 }]);
const controller = new AbortController();
setTimeout(() => controller.abort(), 50);
await assert.rejects(
fetchWithRetry(`${api.url}/j`, {}, { signal: controller.signal }),
(error: unknown) => error instanceof DOMException && error.name === "AbortError",
);
assert.equal(api.hits("/j").length, 1);
});
});
describe("helpers", () => {
it("parses Retry-After as seconds and as an HTTP date", () => {
const now = Date.parse("2026-10-11T12:00:00Z");
assert.equal(parseRetryAfter("120", now), 120_000);
assert.equal(parseRetryAfter("Sun, 11 Oct 2026 12:00:30 GMT", now), 30_000);
assert.equal(parseRetryAfter("soon", now), undefined);
assert.equal(parseRetryAfter(null, now), undefined);
});
it("keeps full jitter within [0, min(cap, base * 2^n)]", () => {
assert.equal(backoffDelay(0, 200, 5_000, () => 0), 0);
assert.equal(backoffDelay(3, 200, 5_000, () => 1), 1_600);
assert.equal(backoffDelay(10, 200, 5_000, () => 1), 5_000);
assert.equal(backoffDelay(2, 200, 5_000, () => 0.5), 400);
});
});import assert from "node:assert/strict";
import { after, before, describe, it } from "node:test";
import { CircuitBreaker } from "../src/circuit-breaker.ts";
import { CircuitOpenError } from "../src/errors.ts";
import { startFlakyServer } from "../src/flaky-server.ts";
import { RateLimiter } from "../src/rate-limiter.ts";
import { createRatesClient } from "../src/rates-client.ts";
let api: Awaited<ReturnType<typeof startFlakyServer>>;
before(async () => (api = await startFlakyServer()));
after(() => api.close());
const noRetries = { retries: 0 };
const rates = { base: "KZT", rates: { USD: 0.0019 } };
describe("circuit breaker", () => {
it("opens after N outages, fails fast, then closes after a successful trial", async () => {
let clock = 0;
const transitions: string[] = [];
const breaker = new CircuitBreaker({
name: "rates-api", failureThreshold: 3, openMs: 30_000, now: () => clock,
onStateChange: (from, to) => transitions.push(`${from}->${to}`),
});
const client = createRatesClient({ baseUrl: api.url, breaker, retry: noRetries });
api.script("/rates", [{ status: 503 }, { status: 503 }, { status: 503 }, { status: 200, body: rates }]);
for (let i = 0; i < 3; i++) await assert.rejects(client.getRates());
assert.equal(breaker.state, "open");
await assert.rejects(client.getRates(), CircuitOpenError);
assert.equal(api.hits("/rates").length, 3); // the 4th call never reached the server
clock += 30_000; // the open period is over: one trial call goes through
const result = await client.getRates();
assert.equal(result.source, "live");
assert.deepEqual(transitions, ["closed->open", "open->half-open", "half-open->closed"]);
});
it("does not count our own 4xx as an outage", async () => {
const breaker = new CircuitBreaker({ name: "strict", failureThreshold: 1 });
const client = createRatesClient({ baseUrl: api.url, breaker, retry: noRetries });
api.script("/rates", [{ status: 401 }]);
await assert.rejects(client.getRates(), /rejected the request: 401/);
assert.equal(breaker.state, "closed");
});
});
describe("degradation", () => {
it("serves the last good answer from the cache during an outage", async () => {
let clock = 1_000_000;
const client = createRatesClient({ baseUrl: api.url, retry: noRetries, now: () => clock });
api.script("/rates", [{ status: 200, body: rates }, { status: 503 }]);
assert.equal((await client.getRates()).source, "live");
clock += 60_000;
const degraded = await client.getRates();
assert.equal(degraded.source, "stale-cache");
assert.equal(degraded.ageMs, 60_000);
assert.deepEqual(degraded.data, rates);
});
it("fails when the cache is too old", async () => {
let clock = 0;
const client = createRatesClient({ baseUrl: api.url, retry: noRetries, maxStaleMs: 1_000, now: () => clock });
api.script("/rates", [{ status: 200, body: rates }, { status: 503 }]);
await client.getRates();
clock += 5_000;
await assert.rejects(client.getRates(), /answered 503/);
});
});
describe("rate limiter", () => {
it("lets a burst through and then spaces requests out", async () => {
const limiter = new RateLimiter({ capacity: 2, ratePerSecond: 10 });
const started = performance.now();
await limiter.acquire();
await limiter.acquire();
assert.ok(performance.now() - started < 20, "the burst is immediate");
await limiter.acquire();
assert.ok(performance.now() - started >= 90, "the third request waits ~100 ms");
});
});
> resilient-fetch@1.0.0 test
> node --test
▶ fetchWithRetry
✔ retries 5xx with exponential delays and returns the success (17.6958ms)
✔ does not retry a 4xx and returns it to the caller (1.1994ms)
✔ waits as long as Retry-After says on 429 (14.2756ms)
✔ gives up at once when Retry-After is longer than allowed (13.248ms)
✔ aborts a hanging attempt by timeout and retries (120.7663ms)
✔ retries a dropped connection (18.6495ms)
✔ does not retry POST without an idempotency key (3.393ms)
✔ retries POST with an idempotency key, sending the same key (12.1761ms)
✔ stops at the total deadline even if attempts remain (262.2019ms)
✔ does not retry when the caller cancels (63.5269ms)
✔ fetchWithRetry (532.1418ms)
▶ helpers
✔ parses Retry-After as seconds and as an HTTP date (0.2605ms)
✔ keeps full jitter within [0, min(cap, base * 2^n)] (0.1896ms)
✔ helpers (0.6801ms)
▶ circuit breaker
✔ opens after N outages, fails fast, then closes after a successful trial (39.2041ms)
✔ does not count our own 4xx as an outage (2.3658ms)
✔ circuit breaker (45.7581ms)
▶ degradation
✔ serves the last good answer from the cache during an outage (10.3849ms)
✔ fails when the cache is too old (31.4225ms)
✔ degradation (42.0438ms)
▶ rate limiter
✔ lets a burst through and then spaces requests out (108.3147ms)
✔ rate limiter (108.5081ms)
ℹ tests 17
ℹ suites 5
ℹ pass 17
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 670.8636$ npm run typecheck
> resilient-fetch@1.0.0 typecheck
> tscThe tests were checked in reverse: remove the idempotency condition and the test about POST without a key fails; ignore Retry-After and the 429 test fails. The hang test waits for a real 100 ms timeout and the deadline test for 250 ms; the other pauses are replaced with recording into an array.
fetchwithout a timeout: a hung service holds requests and connections- Retrying everything, including
400and401 - Automatically retrying
POSTwithout an idempotency key — double payments and orders - Retries without a pause or with the same pause for all clients, no jitter
- Retries at every layer: HTTP client, service, queue — the load multiplies
Retry-Afterignored and the provider limit not respected in advance- The breaker trips on our own
4xxresponses - Stale cached data shown where accuracy matters: payments, stock levels
How this was verified3 · passedchecked 11 October 2026
“Run” means the code was executed in a scratch project or container and the output in the article comes from that run. “Partial” means it was checked against stubs instead of external services. “Not run” means it needs a device, an account or a key; the steps were checked against the official documentation. The verification project files and run logs are kept in the InterLAB archive.