RabbitMQ: a second service on the same queue silently took half the events

On a Monday morning the support team reported orders stuck in "paid" with no shipment created. By noon about 48% of new orders were affected. The fulfilment service logs showed no errors, no retries, no dead-lettered messages. It processed every message it received; it simply received about half as many as usual.

The orders service publishes an order.paid event to RabbitMQ for every paid order. Its publish counter matched the number of orders exactly. The messages left the publisher and arrived at the broker, then roughly half of them never reached fulfilment.

Reproducing it

The management UI showed the queue orders with a deliver rate equal to the publish rate, so the broker was delivering everything. The question was to whom. Listing consumers answered it:

rabbitmqctl list_consumers -p /

Listing consumers in vhost / ...
queue_name  channel_pid                    consumer_tag                         ack_required  prefetch_count  active  arguments
orders      <rabbit@mq-1.1717.4321.0>      ctag-fulfilment-1                    true          20              true    []
orders      <rabbit@mq-1.1717.4388.0>      ctag-fulfilment-2                    true          20              true    []
orders      <rabbit@mq-1.1717.5102.0>      amq.ctag-Xb1c9fQwz0Ll0mY3pQe6hA      true          50              true    []
orders      <rabbit@mq-1.1717.5167.0>      amq.ctag-q8Hn2Lw0t5Vb6ZpTe3Rr9g      true          50              true    []

Two consumers were ours. Two were not. Their connections, visible under the Connections tab, came from the analytics cluster. On Friday, the analytics team had shipped a service that "subscribes to order events" by consuming from the orders queue they found in our docs.

What we ruled out

• Publisher confirms failing. Publisher confirms were enabled and every publish was acknowledged.
• Messages expiring or dead-lettered. The queue had no TTL and the dead-letter queue was empty.
• Fulfilment acking without processing. Its processed counter matched its delivered counter.

Root cause

A RabbitMQ queue delivers each message to one consumer. With four consumers attached, the broker spread messages across all four, weighted by available prefetch capacity. Analytics acknowledged every message it got, which removed it from the queue for good. Fulfilment never saw those orders.

What is happening under the hood

RabbitMQ separates routing from storage. Publishers send to an exchange. The exchange routes copies of the message into zero or more queues according to bindings. Each queue then hands every message to exactly one of its consumers, and deletes it once acknowledged.

So the unit of "who gets a copy" is the queue, not the consumer:

• Several consumers on one queue are competing consumers. They share the work. That is how you scale one service horizontally.
• Several queues bound to one exchange each get their own copy. That is how several services receive the same event.

Our orders service published through the default exchange with routing key orders, which delivers straight into the queue with that name. There was no exchange in the design at all, so the only way for a second team to "subscribe" was to join the queue, and joining a queue means splitting it.

This is also where RabbitMQ and Kafka differ in a way that bites teams moving between them. In Kafka, a topic is a log: messages stay for the retention period no matter who reads them, and each consumer group tracks its own offset, so a new group gets every message without affecting anyone else. A RabbitMQ classic or quorum queue is not a log. A consumed and acknowledged message is gone. The Kafka habit "just add a consumer group" maps to "add a queue and a binding" in RabbitMQ, never to "add a consumer to the existing queue".

Environment

Minimal reproduction

# publish 1000 messages to the default exchange, then run two consumers on the same queue
import pika

conn = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
ch = conn.channel()
ch.queue_declare(queue="orders", durable=True)
for i in range(1000):
    ch.basic_publish(exchange="", routing_key="orders", body=f"order-{i}".encode())
conn.close()

# consumer.py: run twice in parallel, as "a" and "b"
import sys
import pika

name, seen = sys.argv[1], 0
conn = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
ch = conn.channel()
ch.basic_qos(prefetch_count=10)


def on_msg(channel, method, props, body):
    global seen
    seen += 1
    channel.basic_ack(method.delivery_tag)
    if seen % 100 == 0:
        print(name, seen, flush=True)


ch.basic_consume(queue="orders", on_message_callback=on_msg)
ch.start_consuming()

Each consumer ends at roughly 500. Neither sees all 1000.

Who is affected

Any system where a second consumer of an "event" attaches to a queue that already has an owner. It is most common when the publisher uses the default exchange, when queues are named after the event instead of the consuming service, and when teams with a Kafka background integrate with RabbitMQ.

The fix

We introduced a fanout exchange for order events and gave each consuming service its own queue.

# topology.py: declared by each service at startup (idempotent)
import pika


def declare(ch: pika.adapters.blocking_connection.BlockingChannel, service: str) -> str:
    ch.exchange_declare(exchange="orders.events", exchange_type="fanout", durable=True)
    queue = f"orders.{service}"
    ch.queue_declare(queue=queue, durable=True, arguments={"x-queue-type": "quorum"})
    ch.queue_bind(queue=queue, exchange="orders.events")
    return queue

- ch.basic_publish(exchange="", routing_key="orders", body=payload, properties=props)
+ ch.basic_publish(exchange="orders.events", routing_key="order.paid", body=payload, properties=props)

We used a fanout because every service wanted every order event. If services want subsets, a topic exchange with routing keys like order.paid and order.refunded and bindings such as order.* works the same way.

The migration had to lose nothing, so the order mattered:

1. Deploy fulfilment declaring orders.fulfillment bound to orders.events, consuming from both the old orders queue and the new one. Processing is idempotent by order ID, which made double delivery during the switch harmless.
2. Ask analytics to stop consuming orders and declare orders.analytics the same way.
3. Switch the publisher to orders.events.
4. Wait until orders is empty, stop consuming it, and delete it.
5. Replay the 48 hours of affected orders from the orders database into fulfilment.

Alternatives and trade-offs

• RabbitMQ streams. A stream is an append-only log inside RabbitMQ; consumers read with offsets and do not remove messages. It gives Kafka-like replay without a new platform, at the cost of different client semantics.
• Kafka for event distribution. If many teams need the same history and replay, a log is the better model. We kept RabbitMQ because fulfilment needs per-message acknowledgement and dead-lettering.
• Shovel or a forwarding consumer. Copying from one queue into others works but adds a moving part and hides ownership.

How to test the fix

An integration test against a real broker (we use a RabbitMQ container in CI) publishes N events and asserts each service queue received all N:

def test_each_service_gets_every_event(rabbit_channel):
    q1 = declare(rabbit_channel, "fulfillment")
    q2 = declare(rabbit_channel, "analytics")
    for i in range(200):
        rabbit_channel.basic_publish(exchange="orders.events", routing_key="order.paid", body=str(i).encode())
    assert rabbit_channel.queue_declare(queue=q1, passive=True).method.message_count == 200
    assert rabbit_channel.queue_declare(queue=q2, passive=True).method.message_count == 200

python -m pytest -q tests/integration/test_order_topology.py

Confirming in production

After the publisher switched, we compared per-queue publish rates in the management API: orders.fulfillment and orders.analytics each received exactly the orders-service publish count. The daily reconciliation job, which counts paid orders without a shipment, went from about 1,900 to zero and stayed there.

rabbitmqctl list_bindings -p / source_name destination_name routing_key
rabbitmqctl list_queues -p / name messages consumers

Regression guard

• Queues are named <domain>.<consuming-service>, never after the event alone.
• Publishers never use the default exchange for events; a lint check in the shared messaging library rejects exchange="" for event publishing.
• The reconciliation job alerts if paid orders without a shipment exceed 10 for more than 15 minutes.

Security implications

The incident was possible because the analytics credentials had read access to every queue in the vhost. Any service with that access can consume, and therefore delete, another team's messages. We now give each service user permissions only on its own queue plus read on the exchange it binds to:

rabbitmqctl set_permissions -p / analytics \
  '^orders\.analytics$' '^orders\.analytics$' '^(orders\.analytics|orders\.events)$'

The three patterns are configure, write and read. With them, analytics can declare and bind its own queue but cannot consume from orders.fulfillment.

Prevention checklist

• One queue per consuming service; several consumers on that queue only for scaling that service.
• Publish events to a named exchange, never straight to a queue.
• Scope broker permissions per service user.
• Document topology as "exchange plus how to bind", not "queue name".
• Reconcile business counts end to end so silent loss is caught in minutes.

References

• RabbitMQ AMQP 0-9-1 model explained: https://www.rabbitmq.com/tutorials/amqp-concepts
• RabbitMQ consumers and prefetch: https://www.rabbitmq.com/docs/consumers
• RabbitMQ access control: https://www.rabbitmq.com/docs/access-control
• RabbitMQ streams: https://www.rabbitmq.com/docs/streams
• Apache Kafka design, consumer groups: https://kafka.apache.org/documentation/#design