Data Engineering
Bug Report
RabbitMQ: a second service on the same queue silently took half the events
The analytics team started consuming our order events and fulfilment began missing about half of all new orders. Nothing errored. Both services were reading the same RabbitMQ queue, and a queue hands each message to exactly one consumer. We fixed it with an exchange and one queue per service.