Kafka consumer lag grew every evening peak: per-record I/O, idle pods and a partition cap

Our activity pipeline turns raw product events (page views, invoice opens, exports) into enriched rows that power customer dashboards. From about 18:00 to 21:00 UTC every weekday the activity-enricher consumer group fell behind. Dashboards showed data 30 to 40 minutes old, and the lag alert fired nightly:

[FIRING] KafkaConsumerLagHigh group=activity-enricher topic=events.activity lag=1834112 for=15m

Lag drained by around 23:00. Nothing crashed, nothing logged an error, and the platform team had already raised the HPA maximum from 12 to 24 pods without any effect.

Reproducing it

Production traffic peaks around 2,400 records per second on events.activity; off-peak it is closer to 400. We reproduced the evening in staging with the producer perf tool at peak rate:

kafka-producer-perf-test.sh --topic events.activity \
  --num-records 4000000 --record-size 600 --throughput 2400 \
  --producer-props bootstrap.servers=broker-1.example.com:9092 acks=all linger.ms=10

Within five minutes lag grew by roughly 1,800 records per second. Consumption topped out near 600 per second regardless of how many pods we started.

What we ruled out

• Broker capacity. Broker request handler idle ratio stayed above 0.8 and fetch latency was flat. A broker-side bottleneck would slow producers too, and they were fine.
• Rebalances. The group was stable during peaks. (The usual slow-poll rebalance loop has its own write-up; we confirmed no member left the group.)
• Key skew. Lag was spread evenly across all 12 partitions, so this was a capacity problem, not one hot customer.
• CPU. Consumer pods sat at 15 to 20 percent CPU during the peak, which is why the CPU-based HPA barely scaled.

Root cause

Two things together. First, the handler did two blocking network calls per record: a lookup to the profile service (median 14 ms) and a single-row INSERT into PostgreSQL (median 5 ms). At roughly 20 ms per record, one consumer thread handles about 50 records per second. Second, the topic had 12 partitions, and a consumer group can never have more active members than partitions. With concurrency=1 per pod, 12 threads times 50 records per second gives about 600 per second, exactly what we measured. Pods 13 to 24 joined the group, received no partitions and idled.

What is happening under the hood

Consumer throughput is bounded by:

max throughput = active consumer threads x records per second per thread
active consumer threads <= partition count
records per second per thread = 1 / (time spent per record)

So there are only three levers: spend less time per record, process more than one record at a time inside a thread's assignment, or add partitions. Adding consumers beyond the partition count is not a lever. Our pipeline was I/O bound, so the cheapest lever by far was time per record: a network round trip per record is the most expensive thing a consumer can do.

Environment

Minimal reproduction

The original listener:

@KafkaListener(topics = "events.activity", groupId = "activity-enricher")
public void onEvent(ConsumerRecord<String, ActivityEvent> rec) {
    Profile p = profileClient.get(rec.value().accountId());       // ~14 ms
    jdbc.update("INSERT INTO activity (account_id, kind, plan, ts) VALUES (?, ?, ?, ?)",
        p.accountId(), rec.value().kind(), p.plan(), rec.value().ts()); // ~5 ms
}

Run it against the perf-test load above and watch lag:

watch -n 10 "kafka-consumer-groups.sh --bootstrap-server broker-1.example.com:9092 \
  --describe --group activity-enricher | awk 'NR>1 {sum+=\$6} END {print sum}'"

Who is affected

Any consumer doing per-record network I/O, especially when the autoscaler watches CPU. It is easy to miss in development where traffic is a few records per second and latency per record looks fine.