After an OOM kill, our warehouse consumer restarted from its last committed offset and applied a few seconds of stock movements a second time. A quick fix that committed first lost events instead. Storing offsets in the same database transaction as the results fixed both.
After a spot node reclaim, our Spring Cloud Gateway sent traffic to instances that no longer existed for over six minutes. Eureka had entered self-preservation and stopped evicting, while three layers of caches added their own delay. Here is how we cut stale routing to seconds.
Kafka has no message priority, so a marketing blast on the same topic can delay password reset codes by twenty minutes. We compare the designs that actually work: separate topics with dedicated consumers, pausing bulk partitions in one consumer, and broker quotas.
A ledger worker went down with No space left on device while du could account for only 31 GB of a 98 GB volume. A log file deleted by logrotate was still open in the JVM and kept growing. We freed it without a restart and fixed the rotation.
Every evening our activity consumer fell 1.8 million records behind and needed two hours to catch up. Adding pods did nothing. The cause was a per-record HTTP call and single-row insert, capped by twelve partitions. Batching and keyed parallelism cut processing cost 20x.
Our webhook dispatcher threw ConcurrentModificationException a few times a day and silently skipped deliveries the rest of the time. Two bugs shared one list: removal inside a for-each loop and an admin thread editing it concurrently.
We raised a keyed Kafka topic from 6 to 18 partitions to fix lag. Within minutes, order status events arrived out of order and some orders got stuck in the wrong state. The key-to-partition mapping changed under live traffic; here is why and how to resize safely.
Our enrichment consumers rebalanced every few minutes, lag climbed past two million messages and some records were processed twice. A slow external API pushed batch processing past max.poll.interval.ms. How we proved it, fixed the poll loop and made rebalances cheap.