Same idempotency key, different payment amount: our API returned the old charge as success

Our payments API lets partners create charges. Like most payment APIs, it accepts an Idempotency-Key header so a client can safely retry after a timeout without charging twice. A reconciliation run at one partner, a booking platform, found 41 bookings marked "paid" in their system with no matching charge in ours.

For each one, their logs had a successful response from our API with a charge ID. The charge IDs existed, but each belonged to a different booking, with a different amount.

We pulled our idempotency records for those requests:

key                                   | first_seen          | first_body_amount | replayed_at         | replay_body_amount
--------------------------------------+---------------------+-------------------+---------------------+-------------------
bk-2026-10-14-0001                    | 2026-10-14 00:00:03 | 129.00 EUR        | 2026-10-14 00:00:41 | 86.50 EUR
bk-2026-10-14-0002                    | 2026-10-14 00:00:05 | 74.00 EUR         | 2026-10-14 00:01:02 | 212.00 EUR
...

The partner generated idempotency keys from the date plus a counter that reset when their worker restarted. After a restart at midnight, new bookings got keys that had already been used for earlier bookings the same day. Our API saw a known key and replayed the stored response: 201 Created, the old charge ID, the old amount. The new booking's card was never charged.

Our implementation

async def idempotent(request: Request, call_next):
    key = request.headers.get("Idempotency-Key")
    if not key:
        return await call_next(request)

    cached = await redis.get(f"idem:{key}")
    if cached:
        return Response.from_cache(cached)          # replay, no questions asked

    response = await call_next(request)
    await redis.set(f"idem:{key}", response.to_cache(), ex=86400)
    return response

Three problems, visible once we knew what to look for:

1. The stored record had only the response, not what the request was.
2. Keys were global, not per partner, so two partners using the same key format could collide too.
3. Two concurrent requests with the same key could both miss the cache and both run.

What an idempotency key promises

An idempotency key says: "this request is a retry of the one I sent before with the same key; if you already processed it, give me the same result instead of doing it again." The server's promise only makes sense if the retry is the same request. If the body is different, the client is not retrying, it is making a new request with a stale key, and replaying the old response is wrong.

The IETF draft that standardises the Idempotency-Key header says this explicitly: if a request arrives with a key that was already used for a different payload, the server should reject it, and it suggests 422 Unprocessable Content. It also describes 409 Conflict for a request that arrives while the original with the same key is still being processed. Major payment APIs behave the same way and document that reusing a key with different parameters returns an error.

Why the partner's key generation was fragile

The key should be unique per logical operation. Deriving it from something that resets (a counter in memory) or that two operations can share (a timestamp to the second) eventually produces collisions. A random UUID per operation, generated once and stored with the booking so retries reuse it, is the simple correct choice. We document that now; we had not before.

But we cannot make every client get this right. The server has to detect the mistake.

Environment

1. Store a fingerprint of the request with the key

We compute a hash over the parts of the request that define the operation: method, path and a canonical form of the body. JSON is canonicalised (sorted keys, no insignificant whitespace) so semantically identical retries match even if a client library reorders fields.

def fingerprint(method: str, path: str, body: bytes) -> str:
    try:
        canonical = json.dumps(json.loads(body), sort_keys=True, separators=(",", ":"))
    except ValueError:
        canonical = body.decode("utf-8", "replace")
    return hashlib.sha256(f"{method}\n{path}\n{canonical}".encode()).hexdigest()

2. Reserve the key atomically, then complete it

The record now has a state. A new key is reserved with SET NX so two concurrent requests cannot both proceed:

async def idempotent(request: Request, call_next):
    key = request.headers.get("Idempotency-Key")
    if not key:
        return await call_next(request)
    if len(key) > 255:
        return problem(400, "Idempotency-Key too long")

    partner = request.state.partner_id
    rkey = f"idem:{partner}:{key}"                # scoped per partner
    body = await request.body()
    fp = fingerprint(request.method, request.url.path, body)

    reserved = await redis.set(rkey, json.dumps({"state": "processing", "fp": fp}),
                               nx=True, ex=86400)
    if not reserved:
        record = json.loads(await redis.get(rkey))
        if record["fp"] != fp:
            return problem(422, "Idempotency-Key already used with a different request")
        if record["state"] == "processing":
            return problem(409, "A request with this Idempotency-Key is still being processed",
                           headers={"Retry-After": "1"})
        return Response.from_cache(record["response"])

    try:
        response = await call_next(request)
    except Exception:
        await redis.delete(rkey)                  # let the client retry a crash
        raise

    if response.status_code >= 500:
        await redis.delete(rkey)                  # do not pin a transient failure
    else:
        await redis.set(rkey, json.dumps({"state": "done", "fp": fp,
                                          "response": response.to_cache()}), ex=86400)
    return response

What each response means to the client:

We chose to store 4xx results and replay them, because a validation error for the same request will not change on retry, but to release the key on 5xx so the client can try again after a real failure.

3. Do not rely on Redis alone for money

Redis is the fast path. The charge table itself also has a unique constraint on (partner_id, idempotency_key), written in the same transaction as the charge. If a Redis failover loses a record, the database still refuses to create a second charge for the same key, and the handler returns the existing charge after comparing its stored fingerprint.

4. Document key generation

Our API reference now says: generate a random UUID per operation, store it with your own record before the first attempt, reuse it only for retries of that exact request, and expect 422 if you reuse it for anything else. We sent the partner a summary and they switched to UUIDs stored on the booking.

1. Tests for every row of the table above.

async def test_same_key_different_body_is_rejected(client):
    h = {"Idempotency-Key": "k-1"}
    first = await client.post("/v1/charges", json={"amount": 12900, "currency": "EUR"}, headers=h)
    assert first.status_code == 201

    second = await client.post("/v1/charges", json={"amount": 8650, "currency": "EUR"}, headers=h)
    assert second.status_code == 422
    assert await charges_count() == 1

async def test_reordered_json_is_still_a_retry(client):
    h = {"Idempotency-Key": "k-2"}
    a = await client.post("/v1/charges", content='{"amount":500,"currency":"EUR"}', headers=h)
    b = await client.post("/v1/charges", content='{"currency":"EUR","amount":500}', headers=h)
    assert b.status_code == 201 and b.json()["id"] == a.json()["id"]
    assert b.headers["Idempotent-Replayed"] == "true"

2. Concurrency test. Fifty concurrent requests with the same key and body produce exactly one charge; the rest receive either the replayed response or 409.

3. Production monitoring. We track 422 idempotency mismatches per partner. In the first week we saw them from two other partners with similar key bugs, and contacted them before any reconciliation problems.

4. Data repair and security. For the 41 affected bookings, the partner charged the customers again with fresh keys after notifying them, or cancelled where the customer preferred. Scoping keys per partner also closed a subtle information leak: under global keys, a partner who guessed another partner's key format could have received that partner's stored response, including a charge ID and amount.

Takeaways

• An idempotency key identifies one operation. Store a fingerprint of the request with it.
• Same key and different request means a client bug: reject with 422, never replay.
• Reserve keys atomically and answer 409 while the first request is still running.
• Scope keys per client, set a TTL, and back the fast store with a database unique constraint for money.

References: IETF draft: The Idempotency-Key HTTP Header Field, RFC 9110, idempotent methods, Redis SET command.