Performance & Scaling
Bug Report
Kafka consumer lag grew every evening peak: per-record I/O, idle pods and a partition cap
Every evening our activity consumer fell 1.8 million records behind and needed two hours to catch up. Adding pods did nothing. The cause was a per-record HTTP call and single-row insert, capped by twelve partitions. Batching and keyed parallelism cut processing cost 20x.