#consumer-lag

Kafka consumer lag grew every evening peak: per-record I/O, idle pods and a partition cap

Every evening our activity consumer fell 1.8 million records behind and needed two hours to catch up. Adding pods did nothing. The cause was a per-record HTTP call and single-row insert, capped by twelve partitions. Batching and keyed parallelism cut processing cost 20x.