Kafka payment consumer charged customers twice for one OrderPlaced event

Support escalated four tickets in one week from customers who saw two identical card charges for a single order. Finance confirmed the payment provider had two successful captures each time, a few hundred milliseconds apart, both carrying the same order id in their metadata. Every case lined up with a deploy of the payment service.

Our payment worker logs showed the same Kafka record handled by two different pods:

2025-02-11T14:02:07.412Z pod=payments-6c9f7-x2kqp topic=orders.placed partition=5 offset=118244 order=ord_7Q2M charge=ch_test_a19 status=captured
2025-02-11T14:02:07.861Z pod=payments-6c9f7-p8wzn topic=orders.placed partition=5 offset=118244 order=ord_7Q2M charge=ch_test_b07 status=captured

Same partition, same offset, two charges.

Reproducing it

The pipeline: order-service publishes OrderPlaced to orders.placed (12 partitions, keyed by order id). The payment worker, a Python service using confluent-kafka, consumes it, charges the card through the provider's REST API, inserts a row in payments, then commits the offset synchronously.

We reproduced it in staging by producing a steady stream of orders and killing a worker pod with kubectl delete pod --grace-period=0 while it was mid-batch. Roughly one kill in five produced a double charge.

What we ruled out

• Producer duplicates. Order-service runs librdkafka without enable.idempotence, so retried produce requests can write a record twice. That is a real gap and we fixed it later, but in these incidents the duplicate had the same offset, so it was one record read twice, not two records.
• Provider retries. The HTTP client had retries enabled. Provider logs showed two independent API requests from two different source pods, not one request retried.
• An existing dedupe check. The worker did query payments for the order before charging. That check exists and runs; it simply could not see a row that the other pod had not inserted yet.

Root cause

Kafka gives at-least-once delivery to a consumer that commits after processing. When a pod died after charging but before commit(), its partitions were reassigned and the new owner resumed from the last committed offset, so the record came back. Our handler was "check, charge, insert, commit": the check and the insert were separate steps with an external call in between. The new owner ran its check before the dying pod's insert landed, saw nothing, and charged again. Nothing told the provider the second request was a retry, so it happily captured twice.

What is happening under the hood

A consumer's committed offset is the next record it will read after a restart or reassignment. Anything processed after the last commit is processed again by whoever owns the partition next. That window exists with manual commits, with auto-commit, and with any batch size. It cannot be closed from the Kafka side alone, because the side effect (a card charge) lives in another system.

The redelivery itself is legal behaviour. The bug is a handler that is not idempotent: running it twice for the same event produces two effects. Making it idempotent needs two pieces:

1. An atomic claim in our own database keyed by a business identifier (order id or event id), taken before the side effect. A unique constraint does the work, not a SELECT.
2. An idempotency key sent to the provider, derived from the same identifier, so that if we crash after the provider charged but before we recorded it, the retry returns the original charge instead of creating a new one.

Kafka's own exactly-once support (transactions plus read_committed) only covers read-process-write loops where the output also goes to Kafka. An HTTP call to a payment provider is outside it.

Environment

Minimal reproduction

The full repro runs locally with Docker. Infrastructure:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    ports: ["9092:9092"]
  postgres:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: example
      POSTGRES_DB: payments
    ports: ["5432:5432"]

Schema, including a fake provider table so we can count captures:

CREATE TABLE payments (
  order_id   text PRIMARY KEY,
  status     text NOT NULL,
  charge_id  text,
  updated_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE fake_provider_charges (
  charge_id       text PRIMARY KEY,
  order_id        text NOT NULL,
  idempotency_key text UNIQUE,
  created_at      timestamptz NOT NULL DEFAULT now()
);

The broken handler, simplified from production:

# broken_worker.py
import json, time, uuid
import psycopg
from confluent_kafka import Consumer

db = psycopg.connect("postgresql://postgres:example@localhost/payments", autocommit=True)

def provider_charge(order_id, idempotency_key=None):
    time.sleep(0.3)  # network latency to the provider
    charge_id = f"ch_test_{uuid.uuid4().hex[:6]}"
    db.execute("INSERT INTO fake_provider_charges (charge_id, order_id, idempotency_key) "
               "VALUES (%s, %s, %s)", (charge_id, order_id, idempotency_key))
    return charge_id

def handle(event):
    order_id = event["order_id"]
    row = db.execute("SELECT 1 FROM payments WHERE order_id = %s", (order_id,)).fetchone()
    if row:
        return  # looks idempotent, is not
    charge_id = provider_charge(order_id)
    db.execute("INSERT INTO payments (order_id, status, charge_id) VALUES (%s, 'captured', %s) "
               "ON CONFLICT DO NOTHING", (order_id, charge_id))

c = Consumer({"bootstrap.servers": "localhost:9092", "group.id": "payments",
              "enable.auto.commit": False, "auto.offset.reset": "earliest"})
c.subscribe(["orders.placed"])
while True:
    msg = c.poll(1.0)
    if msg is None or msg.error():
        continue
    handle(json.loads(msg.value()))
    c.commit(message=msg, asynchronous=False)

Running two copies of this handler against the same event at the same time, which is exactly what a redelivery after a crash looks like, gives two rows in fake_provider_charges.

Who is affected

Any consumer whose handler performs a non-transactional side effect (HTTP call, email, ledger write in another database) and relies on "we commit after processing" for safety. It is independent of client language and Kafka version. Deploys, autoscaling, OOM kills and network blips all trigger the redelivery window.