Kafka internals for debugging: partitions, replicas, ISR, acks and consumer groups

When a Kafka alert fires at 02:00, nobody has time to reread the design paper. What we need is a short, accurate model of what the cluster is doing so we can map a symptom ("producer timeouts", "lag on two partitions only", "consumers idle after deploy") to a mechanism and then to a command that confirms it. Our platform team runs a handful of KRaft clusters on Kafka 4.1 for a B2B invoicing product, and this is the model we hand new on-call engineers.

The post walks through the storage layout, replication and the in-sync replica set, what acks really promises, and how consumer groups split work. Each section ends with the commands we actually run.

Topics are names, partitions are the real unit

A topic is just a label. Everything that matters operationally happens per partition: a partition is an ordered, append-only log stored as segment files on one broker's disk, with a copy on each replica broker. Ordering is guaranteed only inside a partition. Parallelism is capped by the partition count, because inside one consumer group a partition is read by at most one member at a time.

On disk, a partition directory looks like this:

ls /var/lib/kafka/data/invoices.events-3/
# 00000000000004182733.index
# 00000000000004182733.log
# 00000000000004182733.timeindex
# 00000000000004519021.index
# 00000000000004519021.log
# 00000000000004519021.timeindex
# leader-epoch-checkpoint
# partition.metadata

The number in each file name is the base offset of that segment. Retention deletes whole segments, never single records, which is why a topic with retention.ms=86400000 can still hold data older than a day until the active segment rolls.

Debugging consequences we keep in mind:

• A hot key lands every record on one partition. Lag on one or two partitions while the rest are at zero almost always means skewed keys, not slow consumers.
• Changing the partition count changes which partition a key hashes to. That has ordering consequences and deserves its own runbook.
• Disk usage per broker follows partition placement, not topic size, so one broker can fill up while the others are fine.

Leaders, followers and the ISR

Each partition has one leader replica and zero or more followers. Producers write to the leader; followers fetch from the leader exactly like consumers do. The leader tracks which followers are caught up. That set, including the leader itself, is the in-sync replica set (ISR).

A follower drops out of the ISR when it has not caught up to the leader's log end within replica.lag.time.max.ms (30 seconds by default). It rejoins when it catches up again. In KRaft mode the controller quorum, not ZooKeeper, records the ISR and elects new leaders; ZooKeeper support is gone entirely in the 4.x line.

The high watermark is the offset up to which all ISR members have the data. Consumers can only read below the high watermark. This matters during incidents: if a follower in the ISR is slow, the high watermark stops advancing, and consumers appear "stuck" even though the leader keeps accepting writes.

We check ISR health with:

kafka-topics.sh --bootstrap-server broker-1.example.com:9092 \
  --describe --under-replicated-partitions

kafka-topics.sh --bootstrap-server broker-1.example.com:9092 \
  --describe --topic invoices.events
# Topic: invoices.events  Partition: 3  Leader: 2  Replicas: 2,4,1  Isr: 2,4

Replicas is the assignment, Isr is reality. A partition listed with fewer ISR members than replicas is under-replicated. If the missing broker id is the same across many partitions, the problem is that broker (disk, GC, network), not the topic.

Two broker settings decide what happens when replicas fall behind:

With unclean election disabled, a partition whose ISR is empty has no leader and goes offline rather than silently losing committed data. That is the right default for anything financial.

What acks actually promises

The producer acks setting decides when the leader answers a produce request:

Two details trip people up. First, acks=all means "all members of the current ISR", which could be just the leader if the followers have dropped out. That is why min.insync.replicas=2 matters: below it, the leader rejects writes with NotEnoughReplicasException instead of accepting them with weaker durability. Second, since Kafka 3.0 the Java producer defaults to acks=all and enable.idempotence=true, so many teams have the safe setting without knowing it, and then get confused when ISR shrinkage turns into producer errors.

When producers log this:

org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.

the producer is behaving correctly. The fix is on the broker side: find out why followers left the ISR.

Idempotence is a separate guarantee. With enable.idempotence=true the broker tracks a producer id and per-partition sequence numbers, so a retried batch is not written twice. It removes duplicates caused by producer retries; it does nothing about a consumer processing the same record twice.

Consumer groups and assignment

A consumer group is a set of consumers sharing a group.id. The group coordinator (one broker) assigns partitions to members and stores committed offsets in the internal __consumer_offsets topic. Since 4.0, the new consumer rebalance protocol from KIP-848 is generally available: with group.protocol=consumer, the broker computes assignments incrementally and a single member joining no longer stops the whole group. Classic groups still work and are what most existing clients use.

Rules we rely on when reading lag:

• Consumers beyond the partition count sit idle. Twelve partitions and twenty pods means eight pods doing nothing.
• Lag is per partition: log end offset minus committed offset. A committed offset is the next record to read, not the last one processed.
• Committed offsets only move when the application commits (or auto-commit runs during poll()), so a consumer that processes records but crashes before committing will reread them.

The command we run first:

kafka-consumer-groups.sh --bootstrap-server broker-1.example.com:9092 \
  --describe --group invoice-indexer

# GROUP           TOPIC            PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG    CONSUMER-ID       HOST
# invoice-indexer invoices.events  0          881204          881210          6      indexer-7f9c...   /198.51.100.21
# invoice-indexer invoices.events  3          4519020         4602117         83097  indexer-2b11...   /198.51.100.24

If one partition has most of the lag, look at key skew or one slow consumer instance (the HOST column tells you which pod). If every partition lags evenly, the group as a whole is under-provisioned or a downstream dependency is slow. If CONSUMER-ID is empty, nobody owns that partition right now, which usually means a rebalance is in progress or the group is empty.

Reading common symptoms

Broker metrics worth a dashboard panel each: UnderReplicatedPartitions, OfflinePartitionsCount, ActiveControllerCount (exactly one across the controller quorum), ISR shrink and expand rates, and request handler idle ratio. A steady stream of ISR shrinks and expands on one broker is the earliest warning we get, usually hours before producers see errors.

Leader balance

Leadership drifts over time. After a broker restarts, its partitions get new leaders elsewhere, and unless preferred leader election runs, the restarted broker serves fewer leaders than it should while others carry extra load. auto.leader.rebalance.enable=true (the default) handles this periodically. When we need it immediately, after maintenance for example:

kafka-leader-election.sh --bootstrap-server broker-1.example.com:9092 \
  --election-type preferred --all-topic-partitions

Uneven leader counts show up as one broker with much higher network and request load. It looks like a hardware problem and is not.

Key takeaways

• Think in partitions: ordering, parallelism, lag and disk usage are all per partition.
• Replicas is the plan, Isr is what is actually in sync; compare the two first.
• acks=all is only as strong as min.insync.replicas; producer errors from it are the safety net working.
• Producer idempotence removes retry duplicates on the broker, not duplicate processing in consumers.
• Consumers beyond the partition count are idle; lag on a few partitions points to skew, not capacity.
• In Kafka 4.x the controller quorum owns metadata and ISR changes; there is no ZooKeeper to check.

References

• Apache Kafka documentation, Design: Replication: https://kafka.apache.org/documentation/#replication
• Apache Kafka documentation, Broker and producer configs: https://kafka.apache.org/documentation/#configuration
• Apache Kafka documentation, Basic operations: https://kafka.apache.org/documentation/#basic_ops
• KIP-848, The Next Generation of the Consumer Rebalance Protocol: https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol