Request-latency SLOs do not fit a pipeline of queues, functions and databases where users care about whether an order confirmation arrives in time. How we defined SLIs per hop, an end-to-end freshness SLO, how availability combines across steps, and how we alert on error budget burn.
During a partial outage, our latency dashboard showed p99 at 350 ms while one pod in twelve was taking seconds per request. The panel averaged per-pod p99 values. Percentiles cannot be averaged. How we fixed the query by aggregating histogram buckets first, and what that changed for alerts and SLOs.
A new release started dropping logs in Loki with a 429 and stream limit errors. A logging change had promoted user_id and trace_id to labels, creating hundreds of thousands of streams. How Loki's streams work, how we found the offending labels, and how structured metadata and query-time parsing replaced them.
Every trace for an order ended at the publish step, and the consumer's work appeared as separate, unrelated traces. Our Kafka client was not instrumented, so no traceparent header was written, and the consumer started new root spans. How we injected and extracted context in headers and linked batch consumers correctly.
Our API Lambda logs carried a correlation_id from API Gateway, but the SQS consumer logs had none, so we could not follow a request across the queue. Powertools only reads correlation IDs from the event it is given. How we passed the ID as a message attribute and injected it per record in the consumer.
The contrib distribution of the OpenTelemetry Collector ships hundreds of components we never use. We built our own distribution with the OpenTelemetry Collector Builder: a manifest with only our receivers, processors and exporters, a reproducible image, a smaller attack surface and faster upgrades. The full workflow.
After we changed the scrape interval from 15 to 60 seconds to save money, our request-rate panels went blank in places and alerts flapped. rate() needs at least two samples in its range, and our queries used [1m]. Why the range must be several scrape intervals, how $__rate_interval works, and how we fixed alerts too.
Every team argues about whether to track p95 or p99. The answer depends on traffic volume, fan-out and what the number is for. A practical guide to choosing percentiles for SLOs, alerts, benchmarks and capacity planning, with the sample-size maths, coordinated omission and how to report results honestly.