ETL, streaming, warehouses, data quality bugs.
After an OOM kill, our warehouse consumer restarted from its last committed offset and applied a few seconds of stock movements a second time. A quick fix that committed first lost events instead. Storing offsets in the same database transaction as the results fixed both.
A DuckDB report job queried Parquet files on S3 fine on a laptop and failed with HTTP 403 in its ECS task. Locally DuckDB used a persistent secret from personal credentials; in the container nothing told it to use the task role. How DuckDB secrets and the credential chain work, plus the KMS detail.
Our trading dashboards queried a nightly warehouse and were always hours behind. We moved to Kafka, Flink SQL and Apache Doris: exactly-once ingestion into merge-on-write tables, pre-aggregated minute metrics and partitioning that keeps dashboard queries well under a second.
After a rolling restart of our Kafka cluster, the warehouse system was missing about 300 order events the producer had logged as sent. The producer used acks=1, so the leader acknowledged before followers had the data. How we proved it, and the settings that make acknowledged writes durable.
Our cleaning job filled missing countries and lowercased emails, yet the output table still had nulls and mixed case. Chained assignment wrote to a temporary copy, and the warning that said so had been silenced. We fixed it with .loc and turned on Copy-on-Write.
Most Kafka incidents we get paged for come down to five concepts: partitions, leaders, the in-sync replica set, producer acks and consumer group assignment. This is the mental model we use to read the symptoms and find the cause fast.
Our monthly 'top 3 sellers per region' bonus report left out a sales rep who tied for third, and a rerun picked the other one. ROW_NUMBER breaks ties arbitrarily. We show how to pick RANK, DENSE_RANK or WITH TIES on purpose.
A pipeline you are afraid to re-run is a pipeline that will eventually corrupt data. This guide covers partition overwrite, MERGE upserts, dedup keys, watermarks and why exactly-once is a property you build, not a setting you enable.
The analytics team started consuming our order events and fulfilment began missing about half of all new orders. Nothing errored. Both services were reading the same RabbitMQ queue, and a queue hands each message to exactly one consumer. We fixed it with an exchange and one queue per service.
Scoring 80 million rows with a 600 MB model took over 9 hours on a 20-node Spark cluster. The pandas UDF loaded the model from storage on every Arrow batch, thousands of times per executor. An iterator UDF that loads once per task cut the job to 28 minutes.
Writing a 200-character regex in one go is how parsing bugs are born. We build patterns the way we build code: named parts, explicit groups, alternation that stays contained, lookaheads for extra rules, then extract every field from log lines and key=value strings with tests that keep it honest.
Kafka has no message priority, so a marketing blast on the same topic can delay password reset codes by twenty minutes. We compare the designs that actually work: separate topics with dedicated consumers, pausing bulk partitions in one consumer, and broker quotas.
Our nightly Spark job over 1TB of events finished 1,999 of 2,000 join tasks in minutes, then one task ran for an hour and the executor was killed for exceeding memory. The cause was null and hot join keys that adaptive skew handling did not split. Here is the diagnosis and the fix.
Our support assistant quoted an old refund policy that had been removed from the help center weeks earlier. The ingestion pipeline only ever upserted chunks and never deleted them. We added source-of-truth sync, document versions and a nightly reconciliation job.
Every evening our activity consumer fell 1.8 million records behind and needed two hours to catch up. Adding pods did nothing. The cause was a per-record HTTP call and single-row insert, capped by twelve partitions. Batching and keyed parallelism cut processing cost 20x.
Every trace for an order ended at the publish step, and the consumer's work appeared as separate, unrelated traces. Our Kafka client was not instrumented, so no traceparent header was written, and the consumer started new root spans. How we injected and extracted context in headers and linked batch consumers correctly.
Our internal assistant answered questions about supplier contracts and price lists, and its answers about prices were confidently wrong. Plain text extraction had flattened PDF tables into run-on text. How we found it, switched to layout-aware parsing with tables as chunks, and measured the fix.
Our daily signals disagreed with the exchange chart, and intraday signals appeared and disappeared within the same bar. We resampled UTC millisecond timestamps in local time and included the still-forming last candle. UTC everywhere and using only closed candles fixed both problems.
A new operations dashboard refreshed every 15 minutes and our BigQuery bill jumped by 9,000 USD in a month. Each tile ran SELECT star over a 40 TB events table without a partition filter. Column pruning, required partition filters and a bytes billed cap fixed it.
Our perpetual futures bot reported a monthly PnL 6 percent higher than the exchange's account history. It ignored trading fees on some fills, never recorded funding payments, and valued open positions at last trade price instead of mark price. Building PnL from the income ledger fixed it.
Saving a row and then publishing to Kafka looks harmless until the process dies between the two. We walk through the dual-write bug we shipped, a working outbox with a polling relay you can run locally, the Debezium variant, and how sagas build on it across services.
A churn model reached 0.97 AUC in validation and 0.68 in production. The training set joined customer features from the current snapshot, so it included support tickets and plan downgrades that happened after the prediction date. Point-in-time joins brought validation to an honest 0.81.
Customer CSV uploads failed with KeyError 'email' even though the header clearly said email. Excel had written a UTF-8 byte order mark, and other files arrived in Windows-1252. We now detect the encoding explicitly and normalise headers.
After a scheduler outage, our daily orders DAG caught up two missed runs and both loaded the same day while another day was skipped. The task computed its date from the wall clock instead of the run's data interval.
A moving-average strategy showed strong backtest returns and nothing like it in paper trading. The backtest computed the signal from a candle's close and filled the order at that same close, a price nobody could trade at once the signal was known. Shifting signals to the next bar fixed the simulation.