Why Kafka projects fail: eight failure modes we have seen and how to avoid them

Over the last few years our platform group has helped six product teams adopt Kafka, inherited two clusters from teams that had given up on it, and run a migration off Kafka for one workload where it never belonged. In almost every troubled project the broker software was fine. What went wrong was a decision made in the first month: the use case, the data contracts, the partition plan, the delivery assumptions, or who would run the thing at 03:00.

This post lists the eight failure modes we have seen most often, the symptom that usually reveals each one, and what we do now to avoid it. It is written against Kafka 3.8, where KRaft is production ready and ZooKeeper mode is deprecated.

1. Kafka chosen for a job queue problem

The symptom: the team asks for per-message acknowledgement, per-message retry delays, priority, or "just let any idle worker take the next job". They end up building retry topics, delay topics and custom assignment, and still cannot scale workers beyond the partition count.

Kafka is a partitioned, replayable log. It is excellent when many independent consumers need the same stream, when order per key matters, or when you need to reprocess history. It is a poor fit for a work queue where each task is handled once by whichever worker is free. A queue service (RabbitMQ, SQS, or even a PostgreSQL table with SKIP LOCKED) is simpler for that job.

What we do now: a one-page fit check before any new Kafka use. Do multiple consumers need this data? Do we need replay? Is per-key ordering required? If all three answers are no, we use a queue.

2. No schemas, no contracts

The symptom: a producer team renames a field or changes a type, and three consumers start failing or, worse, silently writing nulls. Nobody knows who consumes the topic.

JSON without a schema feels flexible in week one and becomes a liability by month six. We now require a registered schema (Avro, Protobuf or JSON Schema in a schema registry) for every topic that crosses a team boundary, with a compatibility rule:

curl -s -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "BACKWARD"}' \
  https://schema-registry.example.com/config/orders.placed-value

BACKWARD lets consumers on the new schema read old data, which is what you need to upgrade consumers first. Producers using the registry serializer fail at send time when they try to register an incompatible schema, so the break is caught in CI, not in production. We also record an owner and known consumers for each topic in our service catalogue.

3. Partition count and keys picked by guess

The symptom: lag that no number of consumers fixes, one partition that is always hot, or a later request to "just add partitions" that breaks per-key ordering.

Partition count caps consumer parallelism, and key choice decides both ordering and balance. Both are hard to change later. Our planning sheet for every topic asks:

We size partitions so that one consumer thread per partition handles peak with headroom, round up generously, and avoid keys with heavy skew (a tenant id where one tenant is 40 percent of traffic) or add a sub-key for those.

4. Assuming exactly-once end to end

The symptom: duplicate emails, double-counted metrics, or an occasional double charge after deploys.

Kafka offers idempotent producers and transactions, and Kafka Streams can provide exactly-once processing when input and output are Kafka topics. None of that extends to an HTTP call, an email, or a write to another database. Consumers commit offsets after processing, so a crash in between means the record is processed again.

What we do now: every consumer design states its delivery semantics, and every consumer with external side effects is idempotent, using a dedupe key or a unique constraint. "We use Kafka so it is exactly-once" is not accepted in design review.

5. The development setup never resembled production

The symptom: everything worked on a laptop and the first production week is chaos. Topics appeared with replication factor 1, a broker restart took a service down, or producers saw NotEnoughReplicasException they had never seen before.

Development clusters are usually one broker with auto.create.topics.enable=true and default settings. Production needs replication factor 3, min.insync.replicas=2, topics created explicitly with reviewed configs, and clients that handle leader changes. We now:

• Disable auto topic creation everywhere, including development.
• Create topics through code (Terraform or a topic operator), never by first use.
• Run integration tests against a three-broker cluster in CI once a day, and do a rolling restart during the test to prove clients survive it.

6. Nobody owns operations

The symptom: the cluster runs fine for months, then a disk fills, an under-replicated partition goes unnoticed for days, or a broker upgrade is postponed for two years.

Self-managed Kafka needs real operational ownership: capacity planning, upgrades, rebalancing partitions across brokers, certificate rotation, and monitoring that someone actually watches. Kafka 3.x also brings the ZooKeeper to KRaft migration, which needs planning before ZooKeeper mode is removed in a future major release.

The minimum dashboard we insist on:

If no team is willing to own that list, we recommend a managed Kafka service. Managed is more expensive per byte and much cheaper than an outage.

7. Security added at the end

The symptom: a plaintext cluster reachable from the whole VPC, every service using the same credentials, and a security review that blocks go-live for a month.

Retrofitting security means touching every client. We now start with TLS on all listeners, SASL authentication per service (SCRAM or mTLS), and ACLs that grant each principal only the topics it needs:

kafka-acls.sh --bootstrap-server broker-1.example.com:9093 \
  --command-config admin.properties \
  --add --allow-principal User:billing-service \
  --producer --topic invoices.events

Sensitive fields are either kept out of events or encrypted at the application level, because anyone with read access to a topic can replay its full history for the retention period.

8. No plan for scale and cost

The symptom: the first cluster was sized for one use case, five more teams joined, and now one team's batch backfill slows everyone down while the cloud bill doubles.

Shared clusters need multi-tenancy rules from day one: naming conventions, per-team quotas (producer_byte_rate, consumer_byte_rate), retention limits, and chargeback or at least showback. Tiered storage (early access since 3.6 and still maturing at the time of writing) may help retention costs later, but it does not replace planning. We review each cluster quarterly: bytes in, bytes out, retention by topic, and the five largest topics with their owners.

A checklist before you start

Key takeaways

• Most Kafka failures are design and ownership failures, decided in the first month.
• Use Kafka for streams; use a queue for one-off jobs.
• Schemas with compatibility rules and named owners prevent the most common cross-team breakages.
• Partition count and keys are long-lived decisions; plan them from expected peak and skew.
• Kafka does not make external side effects exactly-once; idempotent consumers do.
• If nobody will own operations and security, pick a managed service.

References

• Apache Kafka documentation: https://kafka.apache.org/documentation/
• Apache Kafka documentation, KRaft and ZooKeeper migration: https://kafka.apache.org/documentation/#kraft
• Apache Kafka documentation, Security: https://kafka.apache.org/documentation/#security
• Confluent Schema Registry, Schema evolution and compatibility: https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html