After an OOM kill, our warehouse consumer restarted from its last committed offset and applied a few seconds of stock movements a second time. A quick fix that committed first lost events instead. Storing offsets in the same database transaction as the results fixed both.
Our trading dashboards queried a nightly warehouse and were always hours behind. We moved to Kafka, Flink SQL and Apache Doris: exactly-once ingestion into merge-on-write tables, pre-aggregated minute metrics and partitioning that keeps dashboard queries well under a second.
After a rolling restart of our Kafka cluster, the warehouse system was missing about 300 order events the producer had logged as sent. The producer used acks=1, so the leader acknowledged before followers had the data. How we proved it, and the settings that make acknowledged writes durable.
Most Kafka incidents we get paged for come down to five concepts: partitions, leaders, the in-sync replica set, producer acks and consumer group assignment. This is the mental model we use to read the symptoms and find the cause fast.
A pipeline you are afraid to re-run is a pipeline that will eventually corrupt data. This guide covers partition overwrite, MERGE upserts, dedup keys, watermarks and why exactly-once is a property you build, not a setting you enable.
The analytics team started consuming our order events and fulfilment began missing about half of all new orders. Nothing errored. Both services were reading the same RabbitMQ queue, and a queue hands each message to exactly one consumer. We fixed it with an exchange and one queue per service.
Kafka has no message priority, so a marketing blast on the same topic can delay password reset codes by twenty minutes. We compare the designs that actually work: separate topics with dedicated consumers, pausing bulk partitions in one consumer, and broker quotas.
Every evening our activity consumer fell 1.8 million records behind and needed two hours to catch up. Adding pods did nothing. The cause was a per-record HTTP call and single-row insert, capped by twelve partitions. Batching and keyed parallelism cut processing cost 20x.
Every trace for an order ended at the publish step, and the consumer's work appeared as separate, unrelated traces. Our Kafka client was not instrumented, so no traceparent header was written, and the consumer started new root spans. How we injected and extracted context in headers and linked batch consumers correctly.
Saving a row and then publishing to Kafka looks harmless until the process dies between the two. We walk through the dual-write bug we shipped, a working outbox with a polling relay you can run locally, the Debezium variant, and how sagas build on it across services.
DataTransfer-Regional-Bytes grew to 11,000 USD a month, behind only EC2 and RDS. Services spread across three zones talked to each other at random, and Kafka consumers read from leaders in other zones. Topology-aware routing and follower fetching cut it by 68 percent.
We designed a multi-tenant payment platform for debt-collection agencies: card and ACH, one-time and recurring plans, settlement offers, disputes and reconciliation. Here is how we kept PCI scope small, why the ledger is double-entry and immutable, and how retries stay safe.
We raised a keyed Kafka topic from 6 to 18 partitions to fix lag. Within minutes, order status events arrived out of order and some orders got stuck in the wrong state. The key-to-partition mapping changed under live traffic; here is why and how to resize safely.
A handful of customers were charged twice for a single order, always around deploys. Kafka delivered the event more than once, which it is allowed to do. The real bug was a check-then-charge consumer with no atomic claim and no idempotency key sent to the payment provider.
Our enrichment consumers rebalanced every few minutes, lag climbed past two million messages and some records were processed twice. A slow external API pushed batch processing past max.poll.interval.ms. How we proved it, fixed the poll loop and made rebalances cheap.
Kafka rarely fails because the broker is buggy. Projects fail because teams pick it for the wrong job, skip schemas, guess at partitions, assume exactly-once, and leave operations and security for later. Here are the eight patterns we keep seeing and what to do instead.