Idempotency Keys for APIs: Safe Retries for Payments and State Changes

A payment request times out after the card network has already authorized the charge. The client sees a network error, retries, and the customer is billed twice. Nothing in the transport tells either side which outcome occurred, so the only reliable fix is to make the operation itself identifiable across attempts. That identifier is the idempotency key.

Why retries duplicate writes

Every network call has three possible outcomes: the request never reached the server, it reached the server and failed before committing, or it committed and the response was lost. A timeout looks identical in all three cases. Retries are therefore unavoidable: mobile clients reconnect, load balancers return 502 and 504 after upstream timeouts, SDKs retry 5xx responses, and queues such as SQS deliver at least once.

A handler that inserts a charge row, decrements inventory, or sends a confirmation email cannot tell that attempt two is the same business event as attempt one. Matching on amount, customer, and minute does not work either, because a customer can legitimately buy the same item twice in a row. The duplicate is not a client bug; it is the default behavior of at-least-once delivery against a non-idempotent handler. Retry amplification makes it worse: five gateway replicas each retrying three times turn one click into fifteen writes.

The idempotency key contract

An idempotency key is a client-generated value that names one logical operation, not one HTTP attempt. The contract has four parts. The client creates the key before the first attempt and sends it on every retry of that operation. The server persists the key with a request fingerprint and the outcome. A repeat with the same key and fingerprint returns the stored outcome without executing the handler again. A repeat with the same key and a different fingerprint is rejected.

Scope is part of the contract. Keys are unique within a tenant and endpoint, not across the entire internet. Two customers may both generate the same UUID; they must never collide in storage or receive each other's responses.

POST /v1/payments HTTP/1.1
Host: api.example.com
Authorization: Bearer sk_live_...
Idempotency-Key: 4f8c2a1e-6b0d-4c7e-9a51-2f3d8c7b1e90
Content-Type: application/json

{"amount":4200,"currency":"usd","customer":"cus_8812","description":"Order 1042"}

# The connection drops before the response arrives. Retry the same operation:
POST /v1/payments HTTP/1.1
Idempotency-Key: 4f8c2a1e-6b0d-4c7e-9a51-2f3d8c7b1e90
Content-Type: application/json

{"amount":4200,"currency":"usd","customer":"cus_8812","description":"Order 1042"}

# Response: 201 Created
# Idempotent-Replayed: true
# {"id":"pay_9F2Kx","status":"succeeded","amount":4200}

The first request committed and the response was lost. The retry returns the stored 201 with the same payment id, so the client can treat both attempts as one operation. The server, not the client, decides whether a key was seen before; a client that merely tries to be careful is not a contract.

Which requests need a key

HTTP idempotency describes the intended effect on the resource, not whether two responses are byte-identical. A repeated DELETE that returns 404 the second time is still idempotent.

PUT replaces the full representation, so repeating it converges. A PUT handler that appends an audit row, sends an email, or increments a counter is not idempotent in effect, even though the method is. PATCH with absolute values such as a status field converges; PATCH with a relative value such as a balance delta does not.

Require keys for any POST that moves money, creates an order or subscription, issues credit, sends a one-time message, or provisions a resource. Require them for internal service calls too. A gateway that deduplicates at the edge does not protect you when an internal worker retries the same job, and it cannot see retries that happen between services. Enforce the requirement on the server: a missing or malformed key returns 400, not a silent pass-through.

Key generation: UUIDv4, ULID, or random token

The format matters less than the properties: at least 122 bits of entropy from a cryptographic source, generated before the first attempt, never derived from the request body, stable across retries, and unique per logical operation.

UUIDv4 is the safest default: every language and database handles it, and its randomness makes collisions negligible. The UUID and ULID generator produces both UUIDv4 and ULID values when you need to test a flow.

ULID encodes a millisecond timestamp plus 80 random bits in Crockford Base32. It sorts by creation time, which helps when scanning key tables and correlating support tickets, but it also leaks the approximate creation time. That is usually acceptable for a payment key and should still be a conscious choice. Never treat the timestamp portion as the uniqueness guarantee.

A random token of 16 bytes encoded as base64url is 22 characters and fits comfortably in a header; hex encoding doubles the length. The random token generator is useful for producing test keys and for verifying that your server accepts the documented alphabet. Whatever the format, use the platform's cryptographic random source, not a seeded PRNG or Math.random().

Never derive the key from the payload. Two legitimate identical payments would collapse into one, and any small change would silently create a second operation. Never use a database sequence, a timestamp, or a per-process counter. Finally, persist the key with the pending operation, whether that is an outbox row, a checkout record, or a worker job, before the first send. Regenerating the key when the SDK retries is the single most common way to defeat the whole design.

Where the key lives: headers and naming conventions

Idempotency-Key is the de facto header name: Stripe uses it, and the IETF httpapi draft standardizes it. Some APIs use X-Idempotency-Key; pick one canonical name, document it, and accept it case-insensitively at the edge. If you support both during a migration, forward both and treat the standard name as authoritative.

Keep identity separate from tracing. X-Request-Id and traceparent identify one HTTP attempt and should change on every retry. An idempotency key identifies the operation and must not change. Using a trace ID as the idempotency key defeats deduplication; using an idempotency key as a trace ID hides the retry chain. Send both headers.

Never put the key in the URL or query string. URLs land in access logs, browser history, proxy caches, and referrer headers, and they are easy to leak in screenshots. Validate the header on the server: 1 to 255 characters, a restricted alphabet such as UUID, base32, or base64url, and an explicit rejection before any row is allocated.

Return the key in the response and add Idempotent-Replayed: true when serving a stored response, so clients and logs can distinguish a replay from a fresh execution. Test the exact behavior with the HTTP request workbench: send a POST, copy the key, resend it with a changed amount, and confirm you receive 422 rather than a second charge. Also confirm that your gateway or service mesh forwards the header; retries inside an SDK or sidecar that strip unknown headers silently remove the protection.

Server-side storage: schema and the unique constraint

Use a durable database table. A cache alone is not an idempotency store, because eviction turns a replay into a second execution. The unique constraint is the concurrency control: two requests with the same key race to insert the same row, and exactly one wins.

CREATE TABLE idempotency_keys (
  id               bigserial PRIMARY KEY,
  tenant_id        uuid        NOT NULL,
  endpoint         text        NOT NULL,        -- 'POST /v1/payments'
  idem_key         text        NOT NULL,
  fingerprint      bytea       NOT NULL,        -- SHA-256 of the canonical request
  state            text        NOT NULL,        -- in_progress | completed | failed | unknown
  response_status  int,
  response_body    bytea,
  response_headers jsonb,
  locked_until     timestamptz,
  created_at       timestamptz NOT NULL DEFAULT now(),
  expires_at       timestamptz NOT NULL,
  UNIQUE (tenant_id, endpoint, idem_key)
);

-- Claim, execute, complete (pseudocode)
row = INSERT INTO idempotency_keys (tenant_id, endpoint, idem_key, fingerprint, state, expires_at)
      VALUES (:tenant, :endpoint, :key, :sha256, 'in_progress', now() + interval '24 hours')
      ON CONFLICT (tenant_id, endpoint, idem_key) DO NOTHING
      RETURNING *;

if row was inserted:
    try:
        result = execute_once(payload)            -- the only place side effects happen
        UPDATE idempotency_keys
           SET state = 'completed', response_status = result.status,
               response_body = result.body, response_headers = result.headers
         WHERE id = row.id;
        return result;
    catch error before any side effect:
        DELETE FROM idempotency_keys WHERE id = row.id;   -- release the key
        raise;

existing = SELECT * FROM idempotency_keys
            WHERE tenant_id = :tenant AND endpoint = :endpoint AND idem_key = :key;

if existing.fingerprint != :sha256:
    return 422 {"error":"idempotency_key_reuse"};
if existing.state == 'completed':
    return replay(existing);                      -- stored status, body, headers
if existing.state == 'in_progress':
    return 409 {"error":"request_in_progress"} with Retry-After: 1;
if existing.state in ('failed', 'unknown'):
    return 409 {"error":"outcome_pending"} and route to reconciliation;

The unique key must include the tenant and the endpoint. A global unique index on the raw client value lets one tenant's key collide with another's and, worse, lets a lookup return a response that belongs to someone else. Even if your keys are globally unique by construction, scope every read by tenant.

Keep the domain write and the idempotency completion in the same transaction where possible. When the side effect is an external call, such as a card authorization or an email, the row must record that the side effect may have happened; the next section covers that case. Cap the stored response at 64 to 256 KB, keep status, content type, and headers, and move larger bodies to object storage with a pointer. Storing a 5 MB invoice PDF in the key table is how that table becomes the largest one in the database. Partition by created_at, index expires_at for cleanup, and add secondary indexes only for the reconciliation queries you actually run.

Request fingerprints and key reuse

On every repeat, compare a fingerprint of the request that originally claimed the key against the current request. The purpose is to detect client bugs and accidental collisions before returning a response that belongs to a different operation. If a client reuses a key after editing the amount from 10 to 100 dollars, replaying the 10 dollar response silently discards the new intent.

Canonicalize before hashing: HTTP method, path, tenant, and the semantic fields of the body. Parse JSON and re-serialize with sorted keys, normalize number formatting, and drop fields that are documented as volatile, such as client timestamps, nonces, and trace IDs. Hash the canonical bytes with SHA-256 and store the digest, not the raw body.

Raw byte comparison fails in practice. {"a":1,"b":2} and {"b":2,"a":1} are the same request, 4200 and 4200.0 are the same amount, and header order is meaningless. Canonicalization removes those false mismatches without weakening the check.

Use 409 Conflict for a request that is still in progress, where retrying after a delay is reasonable, and 422 Unprocessable Content for a completed key reused with a different fingerprint, which is permanent. Some APIs return 409 for both cases, and Stripe returns 400 with a machine-readable idempotency error. The exact number matters less than three properties: it is a 4xx, it carries a distinct error code, and clients can tell retryable from permanent. Never overwrite the original fingerprint or response, and never let a mismatch fall through to normal execution.

Concurrency, replay, and partial failure

A duplicate can arrive in three states, and each needs a different response. A completed key replays the stored result. An in-progress key returns 409 with Retry-After, or the server waits briefly and then replays. A failed or unknown key requires a decision about whether the previous attempt could have committed.

Concurrent duplicates are settled by the unique insert. The loser must not execute. It can poll the row every 50 milliseconds for up to a second and then replay, or return 409 immediately. Unbounded waiting under load exhausts connection pools; a 409 is a valid, honest answer.

Response replay should return the stored status, body, and relevant headers verbatim, plus Idempotent-Replayed: true. Authenticate and authorize the caller before revealing the stored response, because the key is not a capability token. Be careful with responses that contain short-lived signed URLs or one-time tokens: replaying an expired URL is worse than re-executing. Store a regenerable representation for those endpoints or exclude them from response caching.

Re-execution is safe only when the previous attempt provably did not commit. A transaction that rolled back releases the key; a call that timed out after the payment provider captured the charge does not. Partial failure is the hard case, and it has three windows.

The first window is a crash after the domain write commits but before the idempotency row is marked completed. The row remains in progress. Recovery uses a lease: a worker looks up the business record by the idempotency key, which is also stored on the charge or order row, marks the key completed with the real outcome, or marks it failed if no record exists.

The second window is a crash after calling the payment provider but before recording the result. The state is unknown, not failed. Do not re-execute. Reconcile with the provider using a deterministic downstream idempotency key, such as a hash of your key and the step name, so the provider deduplicates, then record the provider's answer.

The third window is a crash after the response was stored. Replay is safe. An idempotency row that has been in progress longer than its lease without a matching business record is an incident, not a cache miss. Alert on it, and never fabricate a success response to unblock the client.

TTL, cleanup, and the boundary of protection

The TTL is the deduplication window, and it must be longer than the longest retry horizon you will ever see. If keys expire after one hour but a queue redelivers after six, the retry creates a duplicate. Stripe expires keys after at least 24 hours; payment APIs commonly keep them for 24 hours to 7 days so that client retries, queue redelivery, provider timeouts, and support-driven replays all stay inside the window.

Expiry is not a correctness boundary for old operations. After a key expires, the same key can in principle be used again, so the generator must make that practically impossible and clients must stop retrying long before expiry. Keep the operation timestamp on the business record so reconciliation can detect an ancient duplicate even after the key row is gone.

Cleanup should run in batches by expires_at, drop old daily partitions, or use a native TTL feature. Native TTL is usually lazy: DynamoDB and MongoDB may keep expired rows readable for minutes or hours, so filter on expires_at in reads. Redis TTL is not durable enough to be the only store; use it as a front cache with the database behind it.

Size the table before you need to. One row per state-changing request, at one million writes per day with a 1 KB response, is roughly 1 GB per day before indexes, or 30 GB for a 30-day window. Compress bodies, cap their size, and archive completed rows separately from the hot dedupe table.

Client retry policy, audit logs, security, and common mistakes

The client side is where most failures are introduced. Generate the key once per logical operation, persist it before the first send, and reuse it for every attempt, including after an app restart. Retry only on ambiguous or transient outcomes: network errors, timeouts, 408, 425, 429, 500, 502, 503, and 504. Use exponential backoff with full jitter, honor Retry-After, and cap total attempts. Never retry 422, because a fingerprint mismatch means the key belongs to another request; create a new key for a genuinely new operation and fix the client bug. On 409, wait and retry with the same key. On 401, refresh credentials and retry with the same key. Mark the operation complete only when a definitive status arrives; a timeout is not a failure.

Situation                          Retry?  Key handling
2xx success                        no      complete the operation
Network error or timeout           yes     same key, backoff with jitter
408, 425, 429                      yes     same key, honor Retry-After
500 with no committed side effect  yes     same key after backoff
502, 503, 504                      yes     same key, upstream may have run
409 key in progress                yes     same key after Retry-After
422 fingerprint mismatch           no      new key for a new operation
400 malformed or missing key       no      fix request, key may be reused
401 or 403                         no*     refresh credentials, same key
404                                no      definitive for this operation

Observability should record one structured event per state transition: tenant, endpoint, key or its hash, fingerprint, state, replayed flag, response status, and latency. Track replay ratio, 409 rate, 422 rate, the age of in-progress rows, and the count of unknown states. A sudden rise in 422 means a client is recycling keys; a rise in 409 means contention or slow handlers. Keep an append-only audit record for payment operations with the same retention as the financial ledger.

Security starts with scope. The key is not an authentication credential and must never grant access on its own. Scope lookups to the authenticated tenant and endpoint; otherwise an attacker who observes a key could read another customer's stored response. Validate length and alphabet before writing, rate-limit key creation, encrypt stored response bodies at rest, and avoid logging raw keys where logs are broadly readable. Sufficient entropy makes keys unguessable, but authorization still comes from the token, not the key.

The common mistakes, roughly in order of damage: generating a new key on every retry; using a request ID or trace ID as the idempotency key; storing a seen flag without the response; omitting the unique constraint; comparing raw request bytes; returning 500 for a duplicate; expiring keys before the retry window closes; deduplicating only at the gateway; not forwarding an idempotency key to the payment provider; and assuming PUT or DELETE makes every side effect safe.