A dependency returns errors for four seconds. Every client notices, every client retries, and the retry wave arrives at the same instant carrying more traffic than the original load. The dependency, which was about to recover, falls over again. Now it is down for forty minutes.
Nothing in that sequence is a bug in the usual sense. Every component did what it was told. The retry logic that was supposed to absorb a transient failure is what converted it into a sustained one.
Retries are the first resilience mechanism most people add and the one most often turned into a weapon. This post is about the three things that separate a retry policy that helps from one that amplifies.
#Failures synchronise clients, and synchronised clients form a herd
Start with why the wave forms at all.
Under normal conditions your clients are desynchronised. They started at different times, their requests arrive at different offsets, and load is smooth in aggregate. Nothing coordinates them.
A failure coordinates them. When a dependency starts erroring, every in-flight caller receives its error at approximately the same moment. That single event aligns clients that were previously spread out. They are now a herd, and they are all holding the same instruction: wait, then try again.
If they all wait the same amount, they all return together. The service sees a burst rather than a smooth curve, and a burst is exactly what it cannot handle while it is still recovering.
The important detail is that the herd persists. Once clients are synchronised, they stay synchronised through every subsequent round unless something actively breaks the alignment.
#Exponential backoff spreads the wave without breaking it up
The standard advice is exponential backoff, and it is not wrong so much as incomplete.
Backoff means the delay grows with each attempt: one second, then two, then four. This reduces total request volume over time, which genuinely helps. What it does not do is desynchronise anybody.
If every client failed at the same instant and every client waits exactly one second, they all return at one second. Then they all return at two. The herd is still a herd; you have simply spaced out its arrivals. The peak is lower than it would have been with immediate retries, but it is still a peak, and it lands while the dependency is at its most fragile.
The fix is randomness, and the canonical treatment is Marc Brooker's Exponential Backoff And Jitter on the AWS Architecture Blog, whose formulations now appear in most AWS SDKs.
Full jitter replaces the delay with a random value drawn from the whole interval:
sleep = random_between(0, min(cap, base * 2 ** attempt))The exponential term still bounds how long a client may wait, but the actual wait is uniformly distributed below it. Clients that failed together return spread across a window rather than at a point.
Decorrelated jitter grows the window based on the previous sleep instead of the attempt number:
sleep = min(cap, random_between(base, prev_sleep * 3))The trade between them is small. Full jitter produces less total work, decorrelated jitter tends to finish slightly sooner. Either is dramatically better than no jitter, and that is the decision that matters. Do not spend an afternoon choosing between them.
Note what full jitter implies: sometimes a client retries almost immediately. That feels wrong when you first read it, and it is the entire point. The distribution matters more than any individual client's politeness.
#Retries multiply through layers, and nobody draws that diagram
This is the failure I would most want you to take away, because it is invisible in any single codebase.
Suppose your gateway retries a failed call three times. It calls a service that also retries three times. That service calls a database client configured, sensibly, to retry three times.
One user request can now produce twenty-seven attempts against the database.
Each layer was configured by someone reasonable who was thinking about their own layer. The multiplication only exists in the composed system, and it is the composed system that falls over. Under load, a dependency that is merely slow gets hit with an order of magnitude more traffic than the request rate suggests, which guarantees it becomes a dependency that is down.
There are two defences and you want both.
Retry at one layer. Pick the layer with enough context to know whether retrying is safe and useful, usually the one closest to the caller's intent, and make the others fail fast. A layer that does not retry should propagate the error immediately rather than dressing it up.
Make retries non-multiplicative with a budget. Which brings us to the mechanism that actually holds under stress.
#A retry budget degrades where a retry count does not
A per-call retry count has no idea what the rest of the system is doing. Three attempts per call is fine when one call in a thousand fails. It is catastrophic when everything is failing, because it means you have tripled your entire outbound load at the exact moment your dependency is least able to serve it.
A retry budget caps retries as a proportion of total requests. Allow retries to consume, say, an additional ten percent of your request volume, and refuse to retry beyond that.
The behaviour under the two conditions is what makes this worth the extra code:
When failures are rare, the budget is never exhausted and every failed call gets its retries. You lose nothing.
When failures are widespread, the budget runs out almost immediately and calls start failing fast instead of retrying. Your outbound load stays flat rather than tripling. The struggling dependency gets a chance to recover, which is the only thing that ends the incident.
That is the property you want: the policy stops being aggressive exactly when aggression is most harmful. A retry count does the opposite.
A workable implementation, with full jitter and a shared budget:
type RetryBudget = {
/** Extra attempts allowed, as a fraction of total calls. */
ratio: number
calls: number
retries: number
}
const budget: RetryBudget = { ratio: 0.1, calls: 0, retries: 0 }
function mayRetry(b: RetryBudget): boolean {
// Refuse once retries exceed their allowed share of traffic. Under broad
// failure this trips almost immediately, which is the point: outbound load
// stays flat instead of tripling against a dependency that is already sick.
return b.retries < b.calls * b.ratio
}
const BASE_MS = 100
const CAP_MS = 20_000
async function withRetry<T>(
operation: () => Promise<T>,
{ attempts = 4, retryable }: {
attempts?: number
retryable: (error: unknown) => boolean
}
): Promise<T> {
budget.calls += 1
let lastError: unknown
for (let attempt = 0; attempt < attempts; attempt++) {
try {
return await operation()
} catch (error) {
lastError = error
// Never retry something that is not going to succeed, and never retry
// something whose repetition is unsafe. That judgement belongs to the
// caller, which is why it is passed in.
if (!retryable(error)) throw error
if (attempt === attempts - 1) break
if (!mayRetry(budget)) throw error
budget.retries += 1
// Full jitter: the exponential term bounds the wait, the random draw
// breaks up the herd. Without the randomness, clients that failed
// together return together, forever.
const ceiling = Math.min(CAP_MS, BASE_MS * 2 ** attempt)
await new Promise((r) => setTimeout(r, Math.random() * ceiling))
}
}
throw lastError
}#Most errors should not be retried at all
A retry policy is only as good as its predicate. The default should be to not retry, with specific exceptions.
Do not retry a client error. A 400 or a 422 will fail identically every time. Retrying it wastes your budget and the dependency's capacity on a request that cannot succeed. The exceptions are 429, which is an explicit instruction to try again later, and 408.
Honour Retry-After when you get one. If a server tells you when to come back, your backoff calculation is not better informed than the server is. Use its value.
Do not retry a non-idempotent write. This is a correctness issue rather than a performance one, and it is the one that produces incidents you cannot fix by scaling.
That last point needs care, because the dangerous case is not an error response. It is a timeout.
When a request to charge a card times out, you do not know what happened. The request may never have arrived. It may have been processed successfully with the response lost on the way back. Those two states are indistinguishable from where you are standing, and only one of them is safe to retry.
Retrying blindly means occasionally charging twice. Not retrying means occasionally failing a charge that would have worked.
The way out is to make the operation safe to repeat. Have the caller generate an idempotency key, send it with the request, and have the receiver store the result against that key. A repeated request with a known key returns the stored outcome rather than performing the work again. Now the ambiguity is resolved by the receiver, which is the only party that knows the truth, and the timeout becomes retryable.
#Stop retrying something that is comprehensively down
Backoff and budgets manage the volume of retries. A circuit breaker addresses a different question: whether to send anything at all.
If a dependency has failed on most recent attempts, the next request is very likely to fail too. A breaker tracks that failure rate and, past a threshold, rejects calls immediately without attempting them. After a cooling period it lets a small number of probes through, and restores normal operation if they succeed.
This helps both sides. Your callers get a fast failure instead of waiting for a timeout, which stops request handlers piling up and consuming your own capacity. The dependency gets near-zero load, which is often the only thing that lets it recover.
A breaker is not a substitute for backoff. It handles sustained failure; jitter handles the transient case and the recovery edge. Systems that survive properly have both.
#The retry you should not have attempted
One last constraint, which is the subject of the next post in this series.
A retry is only worth attempting if there is time left to use the result. If the original caller gave you three seconds and two and a half have elapsed, a retry that takes another second is pure waste: you will produce an answer nobody is waiting for, having consumed capacity that a live request needed.
That means a retry policy needs to know the remaining deadline, not just the attempt number. Backoff, budget and breaker all operate on how much you retry. The deadline determines whether you should be retrying at all.
#References
- Marc Brooker, Exponential Backoff And Jitter, AWS Architecture Blog
- Amazon Builders' Library, Timeouts, retries, and backoff with jitter