Our billing service creates an invoice in PostgreSQL and publishes InvoiceIssued to Kafka so that the email, ledger and analytics services can react. For months the code did exactly that, in that order, and it worked in every test we wrote. Then a node drain killed a pod between the database commit and the Kafka send. Forty-one invoices existed in billing but no customer ever received them, and the ledger never booked the receivables. Nobody noticed for nine days. This is the dual-write problem:...