Request-latency SLOs do not fit a pipeline of queues, functions and databases where users care about whether an order confirmation arrives in time. How we defined SLIs per hop, an end-to-end freshness SLO, how availability combines across steps, and how we alert on error budget burn.
When an agent answer is slow or wrong, logs rarely tell you which step caused it. This guide shows how to trace LLM calls, tool executions and agent runs with OpenTelemetry's GenAI semantic conventions, which attributes to record, and how to handle prompts and privacy.
A field guide to receiving webhooks reliably: verify signatures on the raw body, respond fast, queue the work, deduplicate by event ID, tolerate out-of-order delivery, replay safely and monitor the whole pipeline. With code you can adapt.
A developer added user_id as a label on a request latency histogram to debug slow accounts. Active time series went from 1.2 million to 19 million, our hosted metrics bill quadrupled and the query layer slowed down. We found the culprit, dropped the label and set cardinality limits.
Our LLM bill tripled in one month and nobody could say which feature caused it. This guide covers what we built since: per-feature token accounting, prompt and response caching, model routing, output limits, batch processing and budgets with alerts.
Once more than two services call LLM providers, each one reinvents timeouts, retries and key handling, usually badly. This guide covers what a small internal LLM gateway should do, the timeout and retry rules that work, and how to fail over without making outages worse.
Every trace for an order ended at the publish step, and the consumer's work appeared as separate, unrelated traces. Our Kafka client was not instrumented, so no traceparent header was written, and the consumer started new root spans. How we injected and extracted context in headers and linked batch consumers correctly.
Our API Lambda logs carried a correlation_id from API Gateway, but the SQS consumer logs had none, so we could not follow a request across the queue. Powertools only reads correlation IDs from the event it is given. How we passed the ID as a message attribute and injected it per record in the consumer.
The contrib distribution of the OpenTelemetry Collector ships hundreds of components we never use. We built our own distribution with the OpenTelemetry Collector Builder: a manifest with only our receivers, processors and exporters, a reproducible image, a smaller attack surface and faster upgrades. The full workflow.
A research agent called the same search tool 412 times in one run and spent more overnight than the whole feature spent in a normal week. The tool returned an empty string on errors, the loop had no step limit and nothing tracked spend per run.
Every team argues about whether to track p95 or p99. The answer depends on traffic volume, fan-out and what the number is for. A practical guide to choosing percentiles for SLOs, alerts, benchmarks and capacity planning, with the sample-size maths, coordinated omission and how to report results honestly.
During peak traffic a third of our traces arrived with missing spans, and the ones we needed for incident review were exactly the ones gone. The Collector was refusing data at its memory limit and dropping from a full export queue. Here is how we measured the loss and sized the pipeline.