During a partial outage, our latency dashboard showed p99 at 350 ms while one pod in twelve was taking seconds per request. The panel averaged per-pod p99 values. Percentiles cannot be averaged. How we fixed the query by aggregating histogram buckets first, and what that changed for alerts and SLOs.
Calls from our pods to a payment API had 40 to 200 ms of unexplained latency and CoreDNS was the busiest workload in the cluster. Every lookup of api.payments.example went through five search domains first because of ndots:5. FQDNs with a trailing dot and a lower ndots fixed both.
Enabling dynamic batching in Triton Inference Server doubled throughput for our ranking model and pushed p99 latency from 38 to 85 ms, breaking our SLO. A large queue delay and preferred batch sizes made requests wait for batches that rarely filled at low traffic. Measured tuning fixed both.
Our first voice assistant prototype took two to three seconds to answer, and callers talked over it. Most of the delay was not the language model. How we built a latency budget across audio transport, end-of-turn detection, speech recognition, the LLM and speech synthesis, and what we changed in each stage.
Every team argues about whether to track p95 or p99. The answer depends on traffic volume, fan-out and what the number is for. A practical guide to choosing percentiles for SLOs, alerts, benchmarks and capacity planning, with the sample-size maths, coordinated omission and how to report results honestly.
Why does a long prompt make the first token slow while a long answer makes everything slow? A working model of tokens, prefill, decode and the KV cache, with the arithmetic we use to size GPUs and predict LLM latency and cost.