ETL, streaming, warehouses, data quality bugs.
We raised a keyed Kafka topic from 6 to 18 partitions to fix lag. Within minutes, order status events arrived out of order and some orders got stuck in the wrong state. The key-to-partition mapping changed under live traffic; here is why and how to resize safely.
A handful of customers were charged twice for a single order, always around deploys. Kafka delivered the event more than once, which it is allowed to do. The real bug was a check-then-charge consumer with no atomic claim and no idempotency key sent to the payment provider.
Our enrichment consumers rebalanced every few minutes, lag climbed past two million messages and some records were processed twice. A slow external API pushed batch processing past max.poll.interval.ms. How we proved it, fixed the poll loop and made rebalances cheap.
Our nightly embedding backfill died on the same few documents every night. A handful of chunks were far over the embedding model's token limit because the chunker split by characters and some files were minified JSON. Token-aware chunking and batching fixed it.
Most reporting queries that end up as nested subqueries or app-side loops are one window function away from simple. A practical tour of LAG for gaps, running totals, moving averages and the frame clause defaults that quietly give wrong numbers.
Our local copy of the BTCUSDT order book slowly diverged from the exchange: crossed books, levels that never disappeared, and a best bid above the best ask. We applied depth diffs without buffering, skipped the snapshot alignment rules and never detected gaps. Following the sync procedure fixed it.
We put SNS in front of an SQS queue to fan out order events, and the consumer Lambda started failing with KeyError: 'orderId'. Each SQS message body was now an SNS envelope with our event as an escaped JSON string in Message. How raw message delivery fixes it, and what changes for filtering and attributes.
Kafka rarely fails because the broker is buggy. Projects fail because teams pick it for the wrong job, skip schemas, guess at partitions, assume exactly-once, and leave operations and security for later. Here are the eight patterns we keep seeing and what to do instead.