A producer renamed one event field and a downstream service dropped events for nine days

A customer complained that the loyalty points for three orders had never appeared on their account. Support found the same for hundreds of customers, all for orders placed since the 14th of the previous month. The loyalty service was running, consuming from the orders.completed Kafka topic, and its consumer lag was zero. It had processed every event. It had just not awarded any points for nine days.

The loyalty service's logs were quiet at INFO level. At DEBUG, enabled on one pod, the reason appeared:

DEBUG OrderCompletedHandler - skipping event 7d2a... : missing customer_id
DEBUG OrderCompletedHandler - skipping event 7d2b... : missing customer_id
DEBUG OrderCompletedHandler - skipping event 7d2c... : missing customer_id

Every event was being skipped. The handler:

def handle(event: dict) -> None:
    customer_id = event.get("customer_id")
    if customer_id is None:
        log.debug("skipping event %s : missing customer_id", event.get("event_id"))
        return                      # e.g. guest checkouts have no customer
    award_points(customer_id, event["total"])

The early return existed for guest checkouts, which legitimately have no customer. It turned "the field I need is gone" into "this is a guest order", for every order.

What changed upstream

The orders service's changelog for the 14th included: "Rename customer_id to buyer_id in OrderCompleted event (marketplace terminology)". The events on the topic before and after:

{"event_id": "6c11...", "order_id": "ORD-881201", "customer_id": "C-44120", "total": 59.90, "currency": "EUR"}

{"event_id": "7d2a...", "order_id": "ORD-893377", "buyer_id": "C-44120", "total": 42.00, "currency": "EUR"}

The orders team had checked their own consumers. They did not know the loyalty service read this topic. There was no list of consumers, and nothing in the pipeline compared the new event shape with the old one.

Events are a public API without a contract

In an event-driven system, the producer publishes facts and does not know who consumes them. That decoupling is the point. It also means any change to the event's shape is a change to an API whose clients the producer cannot see.

With plain JSON and no schema, nothing enforces the shape. Kafka stores bytes; it does not care what is in them. Every consumer decodes the bytes according to its own assumptions. A renamed field is, from a consumer's point of view, a removed field plus a new one it has never heard of.

Why the failure was silent

Three things combined to make this invisible for nine days:

1. Lenient parsing. event.get("customer_id") returns None instead of failing.
2. A legitimate-looking branch. "No customer" was a real case, so the missing field was treated as normal data instead of an error.
3. Lag-based monitoring. Consumer lag was zero because the service consumed everything. Lag measures "did you read it", not "did you do anything useful with it".

A strict consumer would have crashed on the first event, which would have been noticed in minutes. Lenient consumers are more robust to additive changes and more dangerous for breaking ones.

Compatibility, precisely

Schema evolution has well-defined compatibility types. For a schema registry such as Confluent Schema Registry, the common ones are:

Renaming a required field breaks both directions: old readers cannot find customer_id, and new readers cannot find buyer_id in old data. Every mode rejects it.

Environment

1. Repair the data

The topic had 14 days of retention, so the events were still there. We deployed a handler that accepts both field names, reset the loyalty consumer group's offsets to the start of the 14th, and reprocessed. Points awarding was already idempotent by order_id, so events processed before the rename were not double-counted.

kafka-consumer-groups.sh --bootstrap-server "$BROKERS" --group loyalty \
  --topic orders.completed --reset-offsets --to-datetime 2026-10-14T00:00:00.000 --execute

2. Register schemas and enforce compatibility

We moved orders.completed to Avro with a schema registry. The producer serialises with the registered schema, and the registry rejects incompatible new versions at registration time, which in our pipeline happens in CI:

{
  "type": "record",
  "name": "OrderCompleted",
  "namespace": "com.example.orders",
  "fields": [
    {"name": "event_id", "type": "string"},
    {"name": "order_id", "type": "string"},
    {"name": "customer_id", "type": ["null", "string"], "default": null},
    {"name": "total", "type": {"type": "bytes", "logicalType": "decimal", "precision": 12, "scale": 2}},
    {"name": "currency", "type": "string"}
  ]
}

curl -s -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "FULL_TRANSITIVE"}' \
  "$REGISTRY/config/orders.completed-value"

We chose FULL_TRANSITIVE for this topic because both old and new consumers run at the same time during deploys, and replays read old data with new code. An attempt to register a version that renames customer_id now fails:

{"error_code":409,"message":"Schema being registered is incompatible with an earlier schema for subject
\"orders.completed-value\", details: [{errorType:'READER_FIELD_MISSING_DEFAULT_VALUE', description:'The
field 'buyer_id' at path '/fields/2' in the new schema has no default value and is missing in the old
schema'...}]"}

Avro also has a supported way to rename: keep the field name and add aliases on the reader side, or add the new field with a default, migrate consumers, and remove the old field in a later version. The registry enforces that each step is compatible.

3. Fail loudly on unexpected shapes

The loyalty handler now distinguishes "no customer" from "cannot read the event":

def handle(event: OrderCompleted) -> None:
    if event.customer_id is None:
        metrics.increment("loyalty.skipped.guest")
        return
    award_points(event.customer_id, event.total)

With a typed, schema-decoded event, a missing field is a deserialisation error, not None. Skipped events are now a metric, and an alert fires when the share of skipped events goes above its normal range (about 4 percent guest checkouts). Nine days of 100 percent skips would have triggered it within the first hour.

4. Make consumers visible

Kafka ACLs now require every consumer group to be registered with an owning team, and our internal catalog lists the consumers of each topic. The producing team gets an automatic list of who to notify when they plan a change, even a compatible one.

1. Compatibility check in CI. The producer's pipeline runs the registry's compatibility test endpoint against the proposed schema before merging:

curl -s -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data @schema.json "$REGISTRY/compatibility/subjects/orders.completed-value/versions/latest"
# {"is_compatible":false}

We tested it with the original rename. The pipeline failed with the registry's error message.

2. Replay test. In staging we replayed the last 14 days of the topic through the current loyalty consumer. Every event deserialised, and the guest-skip rate matched the expected share.

3. Monitoring. The skipped-events alert, plus a new "events processed with effect" metric next to consumer lag on the dashboard. Lag says the consumer is keeping up; the effect metric says it is doing its job.

4. Business follow-up. All 31,000 orders received their points after reprocessing, and support sent affected customers a short note. We confirmed no points were awarded twice by comparing the points ledger with distinct order IDs.

5. Security implications. Registering schemas also documented which events carry personal data, such as customer IDs. We now tag those fields in the schema, and the catalog shows which consumers receive personal data, which our privacy review had been asking for.

Takeaways

• An event's shape is a public API. Producers cannot see their consumers, so the contract must be enforced by tooling.
• Renaming a field is a breaking change in every compatibility mode.
• Use a schema registry with an explicit compatibility mode, and check compatibility in CI.
• Consumers should fail loudly on unreadable events, and monitoring should measure effect, not just lag.

References: Confluent Schema Registry schema evolution and compatibility, Apache Avro specification, schema resolution, Kafka consumer group tool.