Performance & Scaling
Bug Report
Our dashboard p99 looked fine while users timed out: we were averaging per-pod percentiles
During a partial outage, our latency dashboard showed p99 at 350 ms while one pod in twelve was taking seconds per request. The panel averaged per-pod p99 values. Percentiles cannot be averaged. How we fixed the query by aggregating histogram buckets first, and what that changed for alerts and SLOs.