Our traces stopped at the Kafka producer: propagating OpenTelemetry context through message headers

We had spent a quarter adding OpenTelemetry tracing to our order pipeline: the API, the payment service, the database calls. The traces looked great until the point where the order service published an OrderPlaced event to Kafka. Everything that happened after, in the fulfilment and notification consumers, appeared in the tracing backend as separate traces with no parent.

Trace 4bf92f3577b34da6a3ce929d0e0e4736
  POST /orders                       orders-api        212 ms
    INSERT orders                    orders-api          9 ms
    POST payments/charge             payments-api      143 ms
    publish orders.placed            orders-api          4 ms   <- trace ends here

Trace 9a1f0c2e...  (no parent)
  process orders.placed              fulfilment        388 ms
    ...

When a customer complained that confirmation emails took twenty minutes, we could not follow a single order from API call to email.

Checking the message headers

We consumed a message with a CLI client and printed its headers:

kcat -C -b kafka:9092 -t orders.placed -o -1 -e -f 'headers: %h\nvalue: %s\n'

headers: content-type=application/json
value: {"orderId":"o_88123","total":129.9}

No traceparent header. The producer was not writing trace context into the message.

Why the HTTP calls worked but Kafka did not

The services were Java, using the OpenTelemetry Java agent for HTTP and JDBC. The agent instruments the Kafka client too, but the order service published through a small internal library that wrapped the Kafka producer in a custom async executor and sent messages from a thread pool. The agent's instrumentation ran on the sending thread, where no span was current, because the OpenTelemetry context had not been carried over to the executor thread. The produce span was created as a new root, or not at all, depending on the path, and the header it injected (when it did) did not belong to the request's trace.

The Go notification service had no Kafka instrumentation at all: it used a client library without OpenTelemetry support, and nobody had added manual propagation.

What we ruled out

• Sampling. Head sampling at the API was 100% for this test; the problem was structural.
• Header size limits. The traceparent header is under 60 bytes; Kafka headers allow far more.
• Broker configuration. Kafka passes headers through unchanged; they were never written.

Root cause

Trace context was not propagated across the Kafka hop. In the Java producer, the context was lost when work moved to a thread pool without context propagation, so no correct traceparent header was written. In the Go consumer, no code extracted context from headers. Without the header and the extraction, each consumer started a new trace.

Context propagation in messaging

For HTTP, instrumentation injects the W3C traceparent (and optional tracestate and baggage) headers into outgoing requests and extracts them on the server. For messaging, the same headers go into the message headers:

1. The producer, inside the current span, creates a "publish" span and injects context into the message headers.
2. The consumer extracts context from headers when it receives the message.
3. The consumer creates a "process" span whose parent (or link) is the extracted context.

Any step missing breaks the chain.

Thread pools and async code

OpenTelemetry stores the current context in thread-local storage (Java), context.Context (Go), or async context (Node.js). Moving work to another thread, goroutine or callback without passing the context loses it. Java's agent wraps common executors automatically, but custom executors built in unusual ways can escape that.

Parent or link?

For a consumer that processes one message per operation, making the producer's span the parent is natural. For batch consumers that process many messages together, one span cannot have many parents. The OpenTelemetry messaging conventions use span links: the batch processing span links to each message's context.

Environment

Java producer: carry the context into the executor

The internal library now wraps its executor with OpenTelemetry's context-aware wrapper, so tasks run with the caller's context:

import io.opentelemetry.context.Context;

ExecutorService base = Executors.newFixedThreadPool(8);
ExecutorService executor = Context.taskWrapping(base);

public CompletableFuture<RecordMetadata> publish(String topic, String key, byte[] value) {
    return CompletableFuture.supplyAsync(() -> send(topic, key, value), executor);
}

With the context available on the sending thread, the agent's Kafka instrumentation creates a publish span as a child of the request span and injects traceparent into the record headers.

headers: traceparent=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01,content-type=application/json

Go consumer: extract manually

For the Go service, we added a small carrier that adapts Kafka headers to OpenTelemetry's TextMapCarrier interface:

type headerCarrier []kafka.Header

func (c headerCarrier) Get(key string) string {
	for _, h := range c {
		if h.Key == key {
			return string(h.Value)
		}
	}
	return ""
}
func (c headerCarrier) Set(key, value string) {}            // not used on the consumer side
func (c headerCarrier) Keys() []string {
	keys := make([]string, 0, len(c))
	for _, h := range c {
		keys = append(keys, h.Key)
	}
	return keys
}

func handle(msg kafka.Message) {
	ctx := otel.GetTextMapPropagator().Extract(context.Background(), headerCarrier(msg.Headers))
	ctx, span := tracer.Start(ctx, "orders.placed process",
		trace.WithSpanKind(trace.SpanKindConsumer),
		trace.WithAttributes(
			semconv.MessagingSystemKafka,
			semconv.MessagingDestinationName(msg.Topic),
		))
	defer span.End()

	sendConfirmation(ctx, msg.Value)   // downstream calls use ctx, so they join the trace
}

The global propagator is set at startup to W3C Trace Context and Baggage:

otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
	propagation.TraceContext{}, propagation.Baggage{}))

Batch consumer: links instead of parents

The fulfilment service reads up to 200 messages per poll and processes them together for some steps. For those, it creates one span with links to each message:

SpanBuilder builder = tracer.spanBuilder("orders.placed process batch").setSpanKind(SpanKind.CONSUMER);
for (ConsumerRecord<String, byte[]> rec : records) {
    Context msgCtx = propagator.extract(Context.root(), rec.headers(), KafkaHeadersGetter.INSTANCE);
    builder.addLink(Span.fromContext(msgCtx).getSpanContext());
}
Span batchSpan = builder.startSpan();

Per-message work after the batch step uses the extracted context as parent, so each order still has a continuous trace.

Alternatives and trade-offs

• Instrumented client libraries: for Go, switching to a Kafka client with an OpenTelemetry instrumentation package would remove the manual carrier, but meant a client migration.
• Correlation IDs in the payload instead of trace context: useful for logs, but tracing backends do not stitch traces from payload fields.

How to test the fix

An integration test publishes a message inside a test span and asserts that the consumer's span has the same trace id:

curl -s -XPOST https://staging.example.com/orders -d @order.json -H 'content-type: application/json' -D - \
  | grep -i traceparent
# then search the tracing backend for that trace id and expect spans from orders-api, fulfilment, notifications

Confirming in production

Traces for orders now run from the API through fulfilment to the notification email, with the Kafka hop visible as a publish span and a process span. The twenty-minute email delay turned out to be consumer lag in the notification service during a deploy, visible as a long gap between the publish and process spans.

Regression guard

• A contract test in each consumer asserts that processing spans have a parent or links from incoming headers.
• A tracing backend query counts consumer spans without a parent on orders.* topics; an alert fires if it exceeds 1% of messages.

Security implications

Trace context headers are visible to every consumer of the topic. They contain only identifiers, but baggage can carry arbitrary key-value pairs; we do not put personal data in baggage, and the collector drops unknown baggage keys before export. Untrusted producers could forge trace ids, which affects only observability data, not authorization.

Prevention checklist

• Inject trace context into message headers on publish and extract it on consume.
• Make sure async executors and goroutines carry the context.
• Use span links for batch processing.
• Verify propagation with an end-to-end test, not only per-service tests.
• Alert on consumer spans without parents for instrumented topics.

References

• OpenTelemetry, Context propagation: https://opentelemetry.io/docs/concepts/context-propagation/
• OpenTelemetry semantic conventions for messaging spans: https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/
• W3C Trace Context: https://www.w3.org/TR/trace-context/
• OpenTelemetry Java, context and executors: https://opentelemetry.io/docs/languages/java/api/