Data Engineering
Bug Report
Kafka consumer crashed before committing offsets and replayed stock movements twice
After an OOM kill, our warehouse consumer restarted from its last committed offset and applied a few seconds of stock movements a second time. A quick fix that committed first lost events instead. Storing offsets in the same database transaction as the results fixed both.