Here is a configuration that looks careful and is not.
The gateway sets a three second timeout on its call to the orders service. Orders sets a three second timeout on its call to inventory. Inventory sets three seconds on its database query. Every hop is bounded. Every number was chosen deliberately. Nobody was careless.
The user can still wait nine seconds, and the client that gave up after three is going to be billed for all of it.
The mistake is treating a timeout as a property of a call. It is better understood as a share of something the whole request owns, and once you look at it that way the fix is mostly bookkeeping.
#A timeout is relative, a deadline is absolute
The distinction is small and it carries everything.
A timeout is a duration attached to a single operation. "This call may take three seconds." It is evaluated fresh at each hop, which is precisely why it multiplies: each service starts its own three second clock from zero, with no knowledge of how much time has already been spent upstream.
A deadline is a point in time by which the entire request must be finished. "This work is worthless after 14:05:03.250." It is established once, at the edge, and every subsequent hop inherits it.
Under a deadline, the third service in the chain does not get three seconds. It gets whatever remains, which might be four hundred milliseconds, and it needs to know that before it decides what to attempt.
Timeouts compose by addition, which is the wrong operator. Deadlines compose by intersection, which is the right one.
#What each hop actually has to do
Deadline propagation is four rules. None of them is clever.
Establish the deadline once, at the edge. The entry point decides how long the request is worth, from the client's Request-Timeout header, a route-specific policy, or a default. Nothing downstream invents its own.
Compute the remaining budget on arrival. remaining = deadline - now. This is the only number that matters and it shrinks as the request travels.
If nothing remains, fail immediately without starting work. A request that arrives already expired must not be executed. It cannot produce a useful result, and running it consumes capacity that live requests need. Under overload this rule alone is a meaningful share of the recovery.
Bound every downstream call by the remaining budget, minus a margin. Never issue a call with a timeout longer than the time you have left, and leave a little room to serialise a response and get it back up the chain. Setting a three second timeout when you have eight hundred milliseconds left is a lie you tell yourself.
#Send the remaining duration, not the absolute time
The obvious implementation is to put the deadline timestamp in a header and let each service compare it against its own clock. Do not do this.
Clocks on different machines disagree. Not by much when NTP is healthy, but the error is unbounded when it is not, and the failure is silent and asymmetric. A service whose clock runs sixty seconds fast will consider every request expired on arrival and reject all of them. A service running slow will honour deadlines that passed a minute ago.
Send the remaining duration instead. Each hop reads the remaining time, subtracts its own elapsed processing, and passes the smaller number on. Every measurement is then a difference between two readings of the same clock, and clock skew stops mattering entirely.
This is what gRPC does. The grpc-timeout header carries a duration with a unit suffix, so 500m is five hundred milliseconds and 2S is two seconds. It is a duration precisely so that no two machines have to agree on what time it is.
HTTP has no equivalent standard, so pick a header and apply it consistently:
const DEADLINE_HEADER = 'x-request-timeout-ms'
/** Time left for this request, in milliseconds. */
function remainingBudget(request: Request, defaultMs = 5_000): number {
const raw = request.headers.get(DEADLINE_HEADER)
const parsed = raw ? Number.parseInt(raw, 10) : Number.NaN
// An absent or malformed value must not become an unbounded request.
if (!Number.isFinite(parsed) || parsed <= 0) return defaultMs
// An upstream caller can shorten the budget but never extend it. Without
// this clamp any service could grant itself more time by rewriting a header.
return Math.min(parsed, defaultMs)
}Note the clamp. A propagated deadline is a ceiling that only ever descends. If a downstream service can raise it, the guarantee is worthless.
#Carrying it through a request in practice
On the server side, the budget becomes an AbortSignal that every downstream operation observes.
/** Reserve for serialising and returning the response up the chain. */
const RESPONSE_MARGIN_MS = 50
export async function handler(request: Request): Promise<Response> {
const budget = remainingBudget(request)
// Arrived with nothing left. Doing the work would burn capacity to produce
// an answer no caller is still waiting for.
if (budget <= RESPONSE_MARGIN_MS) {
return new Response('Deadline exceeded', { status: 504 })
}
const deadline = AbortSignal.timeout(budget - RESPONSE_MARGIN_MS)
try {
// Both calls share one budget rather than each getting a fresh timeout,
// so two sequential hops cannot exceed what the caller allowed.
const user = await callService('/users/me', request, deadline, budget)
const orders = await callService('/orders', request, deadline, budget)
return Response.json({ user, orders })
} catch (error) {
if (error instanceof DOMException && error.name === 'TimeoutError') {
return new Response('Deadline exceeded', { status: 504 })
}
throw error
}
}
async function callService(
path: string,
incoming: Request,
deadline: AbortSignal,
budget: number
) {
const response = await fetch(`https://internal${path}`, {
// The budget travels with the call, so the next service inherits what is
// left rather than starting a fresh clock of its own.
headers: { [DEADLINE_HEADER]: String(budget - RESPONSE_MARGIN_MS) },
signal: deadline,
})
if (!response.ok) throw new Error(`${path} returned ${response.status}`)
return response.json()
}The single deadline signal shared across both calls is the part worth noticing. Two sequential hops draw down one budget. Give each its own AbortSignal.timeout(budget) and you have rebuilt the addition problem inside a single handler.
Where you need to combine the request deadline with a shorter operation-specific limit, intersect them rather than replacing:
const signal = AbortSignal.any([deadline, AbortSignal.timeout(200)])#Cancellation is the half everyone skips
Bounding how long you wait is only one side. The other is stopping work whose result is no longer wanted.
When a caller times out and disconnects, the work downstream usually continues. The HTTP client has stopped listening, but the database is still executing the query, the service is still holding a connection, and the CPU is still being spent. Nobody will ever read the answer.
Under normal load this is invisible. Under overload it is the mechanism that stops recovery: a growing share of capacity goes to requests that were abandoned, which slows real requests, which causes more of them to be abandoned. The system stays saturated with work for nobody.
So cancellation has to reach the actual resource. For Postgres, statement_timeout is the backstop:
-- Bound at the session level, or per transaction for a single expensive query.
SET LOCAL statement_timeout = '800ms';Better still, derive it from the remaining budget rather than hardcoding it, so a query issued with two hundred milliseconds left is capped at two hundred rather than eight hundred. The pattern generalises: whatever the resource is, the limit you hand it should be computed from the budget, not from a constant somebody chose once.
#What this changes about retries
The previous post in this series covered retry storms, backoff and budgets. Deadlines are the constraint that sits above all of it.
A retry is only worth attempting if enough time remains to complete it and return the result. Check the budget before scheduling the delay, and check it again before the attempt:
const ceiling = Math.min(CAP_MS, BASE_MS * 2 ** attempt)
const delay = Math.random() * ceiling
// The wait plus a realistic attempt must both fit in what is left. Retrying
// into an expired deadline is pure waste: capacity spent on a result that
// arrives after everyone has stopped listening.
if (delay + estimatedCallMs > remainingMs()) throw lastErrorWithout this, exponential backoff eventually schedules a retry that lands after the caller has already given up. You get the cost of the retry and none of the benefit, at exactly the moment the system can least afford it.
#Make the budget observable
Two things are worth recording on every request, and both are cheap.
Log the remaining budget at entry and at each significant hop. When a request fails with a deadline exceeded, this immediately tells you which hop consumed the time, which is otherwise a genuinely hard question to answer from traces alone.
Count requests rejected as already expired on arrival, separately from requests that timed out during processing. They mean different things. Timeouts during processing mean something is slow. Rejections on arrival mean the queue ahead of you is deep, which is a saturation signal and usually the earlier warning of the two.
If that number is non-zero in steady state, your service is not slow. It is oversubscribed, and the timeouts are a symptom rather than the problem.
#The smallest useful version
You do not need a service mesh to get most of this. In order of value:
Establish a deadline at the edge and put the remaining milliseconds in a header. Have every service read it, clamp it, and refuse work that arrives expired. Bound outgoing calls with what remains rather than a constant. Pass the reduced budget onward.
That is perhaps thirty lines spread across a codebase, and it converts a chain of independently reasonable timeouts into something that actually bounds what a user waits.
The nine second request stops being possible, because after the first three seconds there is nothing left to spend.
#References
- gRPC deadlines, which documents the
grpc-timeoutheader and why it carries a duration - Amazon Builders' Library, Timeouts, retries, and backoff with jitter
- PostgreSQL runtime configuration:
statement_timeout