During a partial outage, our latency dashboard showed p99 at 350 ms while one pod in twelve was taking seconds per request. The panel averaged per-pod p99 values. Percentiles cannot be averaged. How we fixed the query by aggregating histogram buckets first, and what that changed for alerts and SLOs.
A new release started dropping logs in Loki with a 429 and stream limit errors. A logging change had promoted user_id and trace_id to labels, creating hundreds of thousands of streams. How Loki's streams work, how we found the offending labels, and how structured metadata and query-time parsing replaced them.
After we changed the scrape interval from 15 to 60 seconds to save money, our request-rate panels went blank in places and alerts flapped. rate() needs at least two samples in its range, and our queries used [1m]. Why the range must be several scrape intervals, how $__rate_interval works, and how we fixed alerts too.