Our notification service sends two very different kinds of messages through Kafka: one-time passcodes, password resets and payment receipts on one side, and newsletters, digests and product announcements on the other. For a year both went through one notifications topic. Then marketing scheduled a 1.2 million recipient campaign at 09:00 on a Monday, and for twenty minutes users waiting for a login code got nothing. The passcodes expired before they arrived.

Kafka did exactly what it was built to do: it delivered records in partition order. There is no priority field, and a consumer cannot skip ahead to the important records. If critical and bulk events share partitions, critical events wait behind bulk ones. This guide covers the designs we evaluated, how each behaves under a burst, and what we run today on Kafka 4.1.

Step zero: decide what "critical" means

Before choosing a mechanism, we wrote a classification table with the product owners. Without it every team argues its events are urgent.

Class Examples Delivery target Can be dropped?
Critical OTP, password reset, payment receipt, security alert p99 under 5 s No
Normal Comment mentions, invoice reminders p99 under 2 min No
Bulk Newsletters, digests, announcements Within 2 h Can be rescheduled

The producer decides the class, because it knows the business meaning. Consumers should never parse payloads to guess priority.

Option 1: one topic per priority, separate consumer groups

The simplest design that works: notifications.critical, notifications.normal, notifications.bulk, each consumed by its own consumer group running in its own deployment.

for t in critical normal bulk; do
  kafka-topics.sh --bootstrap-server broker-1.example.com:9092 \
    --create --topic notifications.$t --partitions 12 --replication-factor 3
done

The producer routes on class:

String topic = switch (msg.priority()) {
    case CRITICAL -> "notifications.critical";
    case NORMAL   -> "notifications.normal";
    case BULK     -> "notifications.bulk";
};
producer.send(new ProducerRecord<>(topic, msg.recipientId(), msg));

Why it works: lag on the bulk topic has no effect on the critical topic's offsets. The critical deployment has its own pods, its own connection pool to the email and SMS providers, and its own autoscaling. A campaign can build millions of records of lag on notifications.bulk while notifications.critical stays at zero.

Where it falls short: the topics still share brokers, disks and network, and the consumers may share downstream dependencies. If the bulk consumers saturate the SMS provider's rate limit, critical messages get throttled at the provider even though Kafka delivered them on time. Isolation has to continue past Kafka: we gave the critical path its own provider API key with its own rate limit.

This is what we run, and it fixed most of the problem on its own.

Option 2: one consumer, pause the bulk partitions

Sometimes one service must handle several priorities, for example a small worker that cannot justify three deployments. The consumer API lets a single consumer subscribe to several topics and stop fetching from some partitions with pause() while it continues to call poll(). Paused partitions keep their assignment and do not trigger a rebalance.

consumer.subscribe(List.of("notifications.critical", "notifications.bulk"));

while (running) {
    ConsumerRecords<String, Notification> recs = consumer.poll(Duration.ofMillis(200));
    handle(recs);

    Set<TopicPartition> bulk = consumer.assignment().stream()
        .filter(tp -> tp.topic().equals("notifications.bulk"))
        .collect(Collectors.toSet());

    if (criticalBacklog(consumer) > 0) {
        consumer.pause(bulk);
    } else {
        consumer.resume(bulk);
    }
}

long criticalBacklog(Consumer<?, ?> c) {
    Set<TopicPartition> crit = c.assignment().stream()
        .filter(tp -> tp.topic().equals("notifications.critical"))
        .collect(Collectors.toSet());
    Map<TopicPartition, Long> end = c.endOffsets(crit);
    return crit.stream().mapToLong(tp -> end.get(tp) - c.position(tp)).sum();
}

Things to get right:

  • endOffsets() is a network call. We cache it for a second instead of calling it on every loop.
  • Records already returned by poll() for bulk partitions are still in your hands; finish them or bound how many you take with max.poll.records.
  • Pausing does not affect the group or the commit position, so resuming is instant.

This gives strict priority within one process. It does not help if the bulk handler itself is slow and blocks the thread; one long bulk batch still delays the next critical poll. Keep bulk batches small in this design.

Option 3: client quotas to protect the cluster

Separate topics stop critical events from queuing behind bulk events. They do not stop a bulk producer from hammering the brokers. A campaign job that writes a million records as fast as it can will raise request latency for everyone on the same brokers, including the critical producer.

Kafka client quotas cap bytes per second per client id or user, enforced by the broker by delaying responses:

kafka-configs.sh --bootstrap-server broker-1.example.com:9092 \
  --alter --entity-type clients --entity-name campaign-sender \
  --add-config 'producer_byte_rate=2097152,consumer_byte_rate=4194304'

kafka-configs.sh --bootstrap-server broker-1.example.com:9092 \
  --describe --entity-type clients --entity-name campaign-sender

The campaign producer must set client.id=campaign-sender for this to apply. When throttled, the client sees higher latency, not errors, and the producer metric produce-throttle-time-avg shows it. We pair quotas with a request percentage quota (request_percentage) for the bulk consumers so they cannot monopolise broker request threads.

Quotas are a fairness tool, not a priority tool. They make bulk traffic slower so that everything else has room, which is exactly what we want during a campaign.

Option 4: separate cluster for the critical path

At very high scale, or when critical events have stricter compliance requirements, some teams run them on a different cluster entirely. Fault isolation is complete, but operating cost doubles. We did not need it: with split topics, separate consumers, quotas and provider-level isolation, the shared cluster was never the bottleneck.

Patterns that look right but fail

Approach What goes wrong
Priority header on one topic, consumer sorts records Sorting only works inside one poll batch; older critical records still sit behind bulk ones in the partition
Dedicated "critical" partitions in one topic via a custom partitioner Consumers in one group still get a mix of partitions; you need custom assignment, and bulk lag still competes for the same threads
Retry failed bulk sends on the main topic Retries re-enter the queue and push critical traffic back further
Scaling bulk consumers aggressively during campaigns Saturates the email/SMS provider, which then throttles critical sends
Letting consumers decide priority from payload Every consumer reimplements the rules, and they drift

For retries, we use separate delay topics per class (notifications.bulk.retry.5m, for example) so failed bulk sends never re-enter a critical path.

Measuring that priority actually holds

We track end-to-end latency per class, from the event timestamp set by the producer to the provider's accepted response:

histogram_quantile(0.99,
  sum by (le, priority) (rate(notification_e2e_seconds_bucket[5m])))

And lag per topic, alerted with different thresholds:

Topic Alert when lag above Paging?
notifications.critical 500 records for 1 min Yes
notifications.normal 20,000 for 10 min Business hours
notifications.bulk No lag alert, only age of oldest record over 2 h No

During the next 1.4 million recipient campaign, critical p99 stayed at 2.1 seconds while bulk lag peaked at 900,000 records and drained in 70 minutes, well inside its target.

Key takeaways

  • Kafka has no record priority; records in a partition are consumed in order, so priority has to come from topology.
  • Separate topics and separate consumer groups per priority class are the default answer.
  • Use pause() and resume() when one consumer must serve several priorities; keep bulk batches small.
  • Client quotas stop bulk producers and consumers from degrading the shared cluster.
  • Isolation has to extend past Kafka to provider rate limits, connection pools and retry topics.
  • Measure end-to-end latency per class; lag alone does not tell you a passcode arrived in time.

References