記録 · Entry

Idempotency keys: how to stop a retry double charging

A timeout never tells you whether the write landed. How an idempotency key settles it, and why the key has to commit in the same transaction as the work.

11 min read

API DesignIdempotencyReliabilityBackendPostgreSQL

A client sends a request to charge a card. Three seconds pass and the connection times out.

You now know exactly one thing: you did not receive a response. You do not know whether the charge happened. The request may never have reached the server. It may have been processed perfectly, with the response lost somewhere on the way back. It may be running right now.

From where the client is standing, those states are indistinguishable, and only one of them is safe to retry. Retrying blindly means occasionally charging a customer twice. Not retrying means occasionally rejecting a payment that already succeeded and telling the customer it failed.

The mechanism that resolves this is an idempotency key. Most implementations of it are subtly wrong in the same three places, and every one of those places only fails under concurrency or a crash, which is to say in production.

#A timeout is not a failure, it is an absence of information

Almost every retry bug starts by treating a timeout as an error. It is not one. An error response is information: the server received your request, made a decision, and told you about it. A timeout is the absence of information, and the correct response to missing information is not to guess.

This is why "do not retry a non-idempotent write" is such common advice, and why it is such unsatisfying advice. It is true, and it leaves you with an operation that fails permanently every time a load balancer hiccups.

The way out is not to avoid the retry. It is to make the operation safe to repeat, so that the ambiguity gets resolved by the only party that actually knows what happened: the server.

An idempotency key is how the client says this is the same request I sent before, not a new one. The server keeps a record of what it did for that key, and a repeat arrival returns the recorded outcome rather than performing the work a second time.

PUT and DELETE are already idempotent by definition in RFC 9110. POST is not, which is exactly why anything that creates a charge, a booking, or an order needs this and a profile update does not.

#The caller generates the key, and generates it once

The key has to come from the client, in the request, as a header:

POST /v1/charges
Idempotency-Key: 0f9c2f8e-9d1c-4a3f-8f0a-6b9e3d2a1c44
Content-Type: application/json

This is the part people get backwards. A server cannot generate the key, because by the time the server sees the second request it has no way to know it is a second request. That is the entire problem being solved.

Two rules follow, and they are the client's responsibility:

Generate the key before the first attempt, not per attempt. A UUID v4 minted inside the retry loop produces a different key every time, which turns your idempotency layer into an expensive no-op. The key belongs to the intent, so generate it when you decide to charge, and hold it for every attempt of that charge.

Do not derive it from the request body. It is tempting to hash the payload and call that the key. Then a customer who genuinely wants to buy the same thing twice in the same minute gets one purchase, and you have built a bug that looks like a feature. The key identifies an attempt, not a shape.

The header name is being standardised as Idempotency-Key in an IETF draft, and Stripe has used that spelling long enough that it is effectively the convention. Use it rather than inventing your own.

#Claim the key before you do the work, not after

Here is the implementation almost everyone writes first:

const existing = await db.findKey(key)
if (existing) return existing.response

const charge = await createCharge(input)
await db.saveKey(key, charge)
return charge

Read, then work, then write. It is correct in a single-threaded universe and wrong in yours.

Two copies of the same request arriving four milliseconds apart both run findKey, both find nothing, and both proceed to createCharge. The window between the read and the write is small, but a duplicate request is not a random event. It is a retry, and retries arrive precisely when the system is slow, which is precisely when that window is widest.

The fix is to make claiming the key an atomic write that either succeeds or tells you someone else got there first. In Postgres that is a unique constraint plus a conditional insert:

create table idempotency_key (
  caller_id    text        not null,
  key          text        not null,
  request_hash text        not null,
  state        text        not null default 'in_progress',
  status       smallint,
  response     jsonb,
  created_at   timestamptz not null default now(),
  primary key (caller_id, key)
);

create index idempotency_key_created_at_idx on idempotency_key (created_at);
insert into idempotency_key (caller_id, key, request_hash)
values ($1, $2, $3)
on conflict (caller_id, key) do nothing
returning state;

If that insert returns a row, you own the key and you are the one doing the work. If it returns nothing, someone else owns it, and you go and read what they recorded. No window, because the database decided, not your application.

Note the composite primary key. Scoping to caller_id matters: two tenants picking the same key is not hypothetical when clients generate keys from a counter, from a request ID, or from anything less careful than a UUID. Without the scope, one customer's retry replays another customer's charge, which is a security incident rather than a bug.

#The result and the work must commit together

This is the failure that survives review, because the code reads correctly.

const charge = await createCharge(input)   // committed
await db.completeKey(key, charge)          // process dies here

The charge is committed. The key record still says in_progress. The retry arrives, sees a claim it cannot interpret, and either waits forever or gives up. Worse, if your recovery logic clears stale in_progress rows, the retry claims the key cleanly and charges the customer again.

The work and the record of the work have to be atomic with respect to each other, which means one transaction:

await db.transaction(async (tx) => {
  const charge = await createCharge(tx, input)
  await tx.query(
    `update idempotency_key
        set state = 'completed', status = $1, response = $2
      where caller_id = $3 and key = $4`,
    [201, charge, callerId, key]
  )
})

Now there are exactly two outcomes. Either the charge exists and the key says so, or neither exists and the key row is still claimable. There is no third state to write recovery code for, and recovery code that runs once a quarter is recovery code that does not work.

#Replay the response, do not just skip the work

A surprising number of implementations detect the duplicate, decline to redo the work, and then return something invented, usually a bare 200 OK or a fresh lookup of the resource.

That breaks the caller in a quieter way. The first attempt would have returned 201 with the created charge, including its ID. The retry returns 200 with something assembled differently, or with a resource that has since been modified. The client took a code path it would never have taken had the network behaved, and you have made a retry observably different from a first attempt, which is the one property idempotency was supposed to guarantee.

Store the status code and the serialised body, and replay both:

if (claimed === null) {
  const row = await db.readKey(callerId, key)
  if (row.state === 'completed') {
    return new Response(JSON.stringify(row.response), {
      status: row.status,
      headers: { 'Idempotent-Replay': 'true' },
    })
  }
}

The Idempotent-Replay header is not required by anything, and it is worth sending. It makes duplicates visible in logs, and the first time you see that number spike you will learn something about a client you did not know.

#The same key with a different body is a caller bug

A client reuses a key from an earlier request and sends a different payload. Perhaps the key was cached too aggressively, perhaps it was derived from a session ID. What should happen?

Not a replay. Replaying the old response tells the caller that a charge for 40 dollars succeeded when they asked for 400, and the mismatch is invisible to them until reconciliation.

Store a hash of the request body alongside the key, compare it on every hit, and reject a mismatch loudly:

const hash = createHash('sha256').update(rawBody).digest('hex')

if (row.request_hash !== hash) {
  return json(422, {
    error: 'idempotency_key_reused',
    message: 'This Idempotency-Key was used with a different request body.',
  })
}

422 rather than 409, because nothing is in conflict. The caller has a bug and the only useful thing you can do is tell them precisely what it is. This check costs one hash per request and has caught more client bugs for me than any amount of schema validation.

#Concurrent duplicates need an answer, not a queue

The interesting case is a retry that arrives while the original is still running. The key is claimed, the state is in_progress, and there is nothing to replay yet.

The tempting move is to block until the original finishes and then return its response. Do not. You have now tied up a connection for the duration of an operation that is already slow, in a system already under enough pressure to be producing timeouts. Under a retry storm that turns into every worker waiting on a request that is itself waiting, which is how a slow endpoint becomes a dead one. If you have read why retries amplify a blip into an outage, this is the same herd wearing a different hat.

Return 409 Conflict immediately and let the client come back:

if (row.state === 'in_progress') {
  return json(409, {
    error: 'request_in_progress',
    message: 'A request with this Idempotency-Key is still being processed.',
  })
}

Fast and honest beats slow and helpful. The client already has a retry policy, and this is exactly the kind of error it exists to handle.

#When the side effect lives outside your database

Everything above assumes the work is a database write, which is the easy case. The moment you call a payment processor, the transaction stops covering the part that matters.

You cannot commit a row and a third-party charge atomically. What you can do is order the writes so that every crash point leaves a state you can reason about:

  1. Claim the key and record the intent, then commit. You now have durable evidence that this request was accepted, before anything irreversible happens.
  2. Call the processor, passing your own idempotency key through to it. Every serious payments API accepts one, and this is what makes step 2 safe to repeat.
  3. Record the outcome and mark the key completed.

If the process dies between 2 and 3, you are left with an intent and no recorded outcome. That is recoverable, because the processor will tell you what it did if you ask with the same key. A reconciliation job that walks in_progress rows older than a few minutes and asks the processor for the truth is not optional here, it is the other half of the design.

The general shape is that you can be atomic with your own storage, and only eventually consistent with anyone else's. Write the code so the gap is a question you can answer rather than a state you have to guess at.

#Keys expire, and the caller has to know the window

Storing keys forever is not free. The rows accumulate, the response bodies are not small, and the table grows without bound.

Pick a retention window, publish it, and enforce it:

delete from idempotency_key
 where created_at < now() - interval '24 hours';

Twenty four hours is the common choice and is comfortably longer than any sane retry policy. What matters more than the number is that it is documented, because a client that reuses a key after expiry does not get an error. It gets a second charge, and it will be entirely your fault for never saying how long the guarantee lasts.

#What it unlocks upstream

The reason to build this is not the endpoint. It is what becomes possible once the endpoint is safe.

Every retry policy runs into the same wall: a timeout on a write cannot be retried, because you do not know what happened. That single restriction is what forces the awkward manual reconciliation, the support tickets about duplicate charges, and the code that fails a payment rather than risk repeating it.

An idempotency key removes the restriction. The ambiguity is resolved by the server, which is the only component that knows the answer, and a timeout on a charge becomes as retryable as a timeout on a read. Your backoff, jitter and retry budget now apply to writes, and the most dangerous class of failure in your system turns into an ordinary one.

That is the trade. A table, a unique constraint, a hash comparison, and one transaction boundary drawn in the right place, in exchange for never again having to decide whether to risk charging someone twice.

#References

Related entries