Ask five engineers which latency percentile matters and you get five answers. "p95, because p99 is noisy." "p99, because that is what users feel." "p99.9, because that is what the big companies use." "The average, because it is easy to explain." Each answer is right for some purpose and wrong for another. The useful question is not which percentile is best, but what the number is for.

This guide covers how we choose percentiles for SLOs, alerts, benchmarks and capacity planning, the maths that decides whether a percentile is even meaningful at your traffic level, and the measurement mistakes that make tail numbers look better than reality.

What a percentile tells you

The p99 latency is the value below which 99% of requests completed. One request in a hundred was slower. It says nothing about how much slower; a p99 of 400 ms is compatible with the slowest 1% taking 450 ms or 30 seconds. That is why mature dashboards show several percentiles and the maximum, or a full heatmap.

The average is the least useful latency statistic. Latency distributions are skewed, with a long tail, and the mean is pulled around by a few extreme values while describing no actual request.

Percentiles and how many users feel them

A single user session usually involves many requests: page loads, API calls, asset fetches. If a page needs 20 backend calls and each has an independent 1% chance of being slower than p99, the chance that at least one of the 20 is slow is:

1 - 0.99^20 = 0.18

18% of page loads hit at least one p99-slow call. Similar maths applies to services that fan out to many backends: if one request calls 100 shards in parallel and waits for all, its latency is roughly the p99 of the shards, almost every time.

Calls per user action Share of actions hitting a p99-slow call Hitting a p99.9-slow call
1 1% 0.1%
10 9.6% 1%
50 39% 4.9%
100 63% 9.5%

So the more calls per user action, the further into the tail you need to look. For a backend service called dozens of times per page, p99.9 can matter more than p99 to the user experience.

Is the percentile meaningful at your volume?

A percentile needs enough samples behind it. To estimate p99 you need well over 100 requests in the window, because by definition only 1% of them describe the tail. A rough guide:

Percentile Requests in window for a rough estimate For a stable estimate
p95 100+ 1,000+
p99 1,000+ 10,000+
p99.9 10,000+ 100,000+

A service doing 5 requests per second gets 1,500 requests in five minutes, enough for a stable p95 and a rough p99, not for p99.9. Low-traffic services with p99.9 alerts generate noise: the "p99.9" is just the slowest two or three requests.

Two consequences:

  • Low-traffic services should use p95 or p99 over longer windows, or slow-request ratios over hours.
  • High-traffic services can use p99.9 with short windows.

Choosing by purpose

SLOs

For SLOs, a percentile threshold is awkward to compute and explain. A better formulation is a ratio: "99% of checkout requests complete within 800 ms, measured over 28 days." That is a p99 target, expressed as good events over total events, which can be summed across time and instances and turned into an error budget.

Pick the percentile by user impact and fan-out: user-facing endpoints with one main call per action often use 95% or 99%; internal services called many times per action use 99% or 99.9%.

Alerts

Alerts need signals that are both meaningful and stable. Alerting on p99.9 of a low-traffic service will page people for noise. We alert on the burn rate of the latency SLO (how fast the error budget is being consumed), with multiple windows, and use percentiles on dashboards for diagnosis.

Benchmarks

When comparing two versions or two configurations, report several percentiles and the maximum, and the number of requests. A benchmark that shows only p95 hides the tail, where regressions in garbage collection, locks and retries usually appear.

Capacity planning

Latency typically stays flat as load grows, then rises sharply as a resource saturates. Track p99 against load in load tests to find the knee of the curve, and plan capacity to stay comfortably below it. Average latency often barely moves until it is too late.

Measure honestly: coordinated omission

Many load tools send a request, wait for the response, then send the next. If the server stalls for two seconds, the tool sends nothing during that time, so it records one slow request instead of the dozens that real users, arriving at their own pace, would have experienced. The stall's impact on the distribution is understated massively. This is called coordinated omission.

To avoid it:

  • Use an open model load generator, which sends requests at a fixed arrival rate regardless of responses. In k6, that is the constant-arrival-rate or ramping-arrival-rate executor.
  • Measure latency from the intended send time, not the actual send time, if your tool supports it.
// k6: open model, 500 requests per second regardless of response time
export const options = {
  scenarios: {
    steady: {
      executor: "constant-arrival-rate",
      rate: 500, timeUnit: "1s", duration: "10m",
      preAllocatedVUs: 200, maxVUs: 2000,
    },
  },
};

Measure in the right place

Server-side histograms measure from when the application starts handling the request. They miss time spent in load balancer queues, TLS handshakes, connection pool waits on the client, and network. For user-facing SLOs, measure as close to the user as possible: at the load balancer or with real user monitoring in browsers and apps. Use server-side percentiles to locate where time goes.

Reporting latency

A latency report that people can trust includes:

  • the percentiles (p50, p95, p99, p99.9 if volume allows) and the maximum;
  • the number of requests behind them and the time window;
  • where it was measured;
  • for benchmarks, the load model (open or closed) and the request rate.

A sentence like "p99 dropped from 420 ms to 310 ms over 1.2 million requests per hour at the load balancer, same traffic mix" is useful. "Latency improved 25%" is not.

Common mistakes

Mistake Why it hurts Do this instead
Averaging percentiles across pods or windows Produces a meaningless number Aggregate histograms, then compute quantiles
p99.9 alerts on low-traffic services Pages for two slow requests Longer windows or slow-request ratios
Reporting only p95 in benchmarks Hides tail regressions Report several percentiles and max
Closed-model load tests Coordinated omission hides stalls Open-model, fixed arrival rate
Server-side only for user SLOs Misses queues and network Measure at the edge or in the client

Key takeaways

  • Choose percentiles by purpose and by how many calls make up a user action.
  • Make sure the window holds enough requests for the percentile you use.
  • Express latency SLOs as ratios of fast requests, and alert on burn rate.
  • Use open-model load tests to avoid coordinated omission.
  • Report percentiles with volume, window and measurement point.

References