When an agent answer is slow or wrong, logs rarely tell you which step caused it. This guide shows how to trace LLM calls, tool executions and agent runs with OpenTelemetry's GenAI semantic conventions, which attributes to record, and how to handle prompts and privacy.
Every trace for an order ended at the publish step, and the consumer's work appeared as separate, unrelated traces. Our Kafka client was not instrumented, so no traceparent header was written, and the consumer started new root spans. How we injected and extracted context in headers and linked batch consumers correctly.
The contrib distribution of the OpenTelemetry Collector ships hundreds of components we never use. We built our own distribution with the OpenTelemetry Collector Builder: a manifest with only our receivers, processors and exporters, a reproducible image, a smaller attack surface and faster upgrades. The full workflow.
During peak traffic a third of our traces arrived with missing spans, and the ones we needed for incident review were exactly the ones gone. The Collector was refusing data at its memory limit and dropping from a full export queue. Here is how we measured the loss and sized the pipeline.