Orders missing after a Kafka broker restart: acks=1 and an unclean leader handover lost writes

The warehouse team noticed that a few hundred orders from a Tuesday afternoon never reached their pick lists. The order service had published an OrderPlaced event for each of them, and its logs showed every send as successful, with partition and offset:

INFO producer: sent OrderPlaced order=o_771203 topic=orders.placed partition=7 offset=18823410

But consuming partition 7 from that offset did not return order o_771203:

kcat -C -b kafka-1:9092 -t orders.placed -p 7 -o 18823405 -c 10 -f '%o %k\n'

18823405 o_771198
18823406 o_771199
18823407 o_771200
18823408 o_771211     <- offsets 18823408-18823420 now hold different orders
...

The offsets the producer reported existed, but they contained different messages. The original messages at those offsets were gone.

What happened at that time

The platform team had run a rolling restart of the Kafka brokers that afternoon to apply a configuration change. The broker that led partition 7 restarted at 14:02:11. The missing orders were all sent between 14:02:10 and 14:02:12, to partitions led by that broker.

The producer configuration

acks=1
retries=3
enable.idempotence=false
linger.ms=5

And the topic:

kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic orders.placed

Topic: orders.placed  PartitionCount: 12  ReplicationFactor: 3  Configs: min.insync.replicas=1

What we ruled out

• Consumer offset problems. We read the partition directly from the broker, independent of consumer groups.
• Retention deleting data. Retention was seven days; the data was hours old.
• The producer not sending. It received acknowledgements with offsets from the broker.

Root cause

With acks=1, the partition leader acknowledges a write as soon as it is in its own log, before followers replicate it. When the leader was stopped during the rolling restart, a few hundred acknowledged messages had not yet reached the followers. A follower became the new leader without them. When the old leader came back, it truncated its log to match the new leader, discarding those messages, and new messages were written at the same offsets. The producer had already been told the writes succeeded, so nothing retried.

What acks means

acks=all waits for the current in-sync replica set (ISR), not for every replica. If the ISR shrinks to only the leader, acks=all behaves like acks=1. That is what min.insync.replicas prevents: with min.insync.replicas=2, a write with acks=all fails with NotEnoughReplicas unless at least two replicas have it, so an acknowledged write always exists on at least two brokers.

Why a controlled restart still lost data

A rolling restart moves leadership away from the broker being stopped, but our restart script stopped brokers without waiting for controlled shutdown to complete, and with acks=1 there is always a window where the leader has data the followers do not. Any leader change, controlled or not, can lose messages in that window.

Defaults changed

Since Kafka 3.0, producer defaults are acks=all and enable.idempotence=true. Our producer was configured explicitly years earlier with acks=1 for "performance", and that explicit setting overrode the safer defaults after upgrades.

Who is affected

Producers with explicit acks=1 or acks=0, topics with min.insync.replicas=1, and clusters where unclean.leader.election.enable=true. Clusters running rolling restarts, broker replacements or experiencing network partitions will hit the window sooner or later.

Environment

Producer: acks=all and idempotence

acks=all
enable.idempotence=true
retries=2147483647
delivery.timeout.ms=120000
max.in.flight.requests.per.connection=5
linger.ms=5

Idempotence lets the producer retry safely: the broker deduplicates retried batches using producer ids and sequence numbers, so retries after a timeout do not create duplicates, and ordering per partition is preserved with up to five in-flight requests.

The send result must be checked, and failures handled, because with stricter settings sends can now fail when the cluster cannot guarantee durability:

producer.send(record, (metadata, ex) -> {
    if (ex != null) {
        log.error("publish failed, will retry via outbox", ex);
        outbox.markFailed(record.key());
    }
});

The order service writes events to a transactional outbox table first (described in an earlier post), so an event that cannot be published is retried later rather than lost.

Topic: min.insync.replicas=2

kafka-configs.sh --bootstrap-server kafka-1:9092 --alter --entity-type topics \
  --entity-name orders.placed --add-config min.insync.replicas=2

With a replication factor of 3 and min.insync.replicas=2, the topic tolerates one broker being down while still accepting writes, and every acknowledged write is on at least two brokers.

Broker: no unclean elections

unclean.leader.election.enable=false     # the default; verified explicitly

An unclean election lets an out-of-sync replica become leader, which loses data by design. Keeping it disabled trades availability for durability on critical topics.

Rolling restart procedure

The restart runbook now waits until each broker has no under-replicated partitions before moving to the next:

until [ "$(kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --under-replicated-partitions | wc -l)" -eq 0 ]; do
  sleep 10
done

Recovering the lost events

The outbox table still had every event. A replay job republished the 300 missing events, identified by comparing outbox entries with what consumers had processed. The warehouse consumer is idempotent by order id, so replaying was safe.

Alternatives and trade-offs

• Higher latency: acks=all adds one replication round trip per batch, a few milliseconds in our cluster. Batching with linger.ms absorbed most of it.
• Availability: with min.insync.replicas=2 and two brokers down, writes fail. For orders, failing is better than silently losing data; the outbox absorbs the outage.

How to test the fix

On a staging cluster, produce continuously with sequence numbers, kill the leader of a partition with kill -9, then verify no acknowledged sequence number is missing:

./produce-seq.sh --topic durability-test --acks all &
kill -9 "$(pgrep -f 'kafka.Kafka' | head -1)"    # on the leader broker
./verify-seq.sh --topic durability-test          # reports gaps among acknowledged sequence numbers

With the old settings, the test showed gaps on every run; with the new ones, none in 50 runs.

Confirming in production

The next rolling restart, two weeks later, produced brief NotEnoughReplicas retries in producer logs while a broker was down, and zero lost messages: consumer counts matched outbox counts exactly.

Regression guard

• A configuration test fails if any production producer sets acks other than all or disables idempotence.
• Topic creation through infrastructure code enforces min.insync.replicas=2 for replication factor 3.
• An alert on under-replicated partitions and on NotEnoughReplicas errors.

Security implications

None directly. Silent data loss in an order pipeline is an integrity problem with financial impact; the outbox and replay also give an audit trail of every event published.

Prevention checklist

• Use acks=all with idempotence for any data you cannot lose.
• Set min.insync.replicas=2 with replication factor 3.
• Keep unclean leader election disabled for critical topics.
• Wait for zero under-replicated partitions between broker restarts.
• Keep a source of truth (outbox) so events can be replayed.

References

• Apache Kafka, Producer configs (acks, enable.idempotence): https://kafka.apache.org/documentation/#producerconfigs
• Apache Kafka, Topic configs (min.insync.replicas): https://kafka.apache.org/documentation/#topicconfigs_min.insync.replicas
• Apache Kafka, Replication design: https://kafka.apache.org/documentation/#replication
• Apache Kafka, Graceful shutdown: https://kafka.apache.org/documentation/#basic_ops_restarting