Our dashboard p99 looked fine while users timed out: we were averaging per-pod percentiles

A Tuesday afternoon incident began with support tickets, not alerts: some customers reported that the search page "spins for ages". On our latency dashboard, the search API's p99 was 350 ms, comfortably under the 800 ms alert threshold. The on-call engineer spent twenty minutes looking elsewhere, because the graph said search was fine.

It was not fine. One of the twelve search pods was on a node with a failing disk; its requests took 3 to 8 seconds. About 8% of all requests went to that pod.

The panel's query

avg(
  histogram_quantile(0.99,
    sum by (pod, le) (rate(http_request_duration_seconds_bucket{service="search"}[5m]))
  )
)

It computes p99 per pod, then averages twelve p99 values. Eleven pods at about 120 ms and one at about 3,000 ms average to roughly 360 ms. The number looks like a latency, but it describes no actual request.

The correct number

Aggregating the histogram buckets across all pods first, then computing the quantile:

histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{service="search"}[5m]))
)

{}  4.6

The real p99 across all requests was 4.6 seconds. With 8% of traffic on the slow pod, more than 1% of all requests were slow, so the overall p99 sat inside the slow pod's distribution.

What we ruled out

• Missing data from the slow pod. Its series were present; it was included in the average.
• Histogram bucket layout. Buckets went up to 10 seconds, enough to represent the slow requests.
• Scrape interval and range. The rate() range was 5 minutes with a 30-second scrape, which is fine.

Root cause

The dashboard averaged per-pod 99th percentiles. Percentiles are not additive: the average of several p99 values is not the p99 of the combined requests, and it systematically hides a minority of slow instances. The alert used the same expression, so it never fired.

Why percentiles cannot be averaged

A percentile is a position in a sorted list of observations. Averaging positions from different lists does not give a position in the combined list. Consider two pods:

The average of p99 values is 1,610 ms. The true p99 over 10,000 requests is around 3 seconds, because about 500 requests, 5% of the total, are slow. With different traffic splits, the average can be far too low or far too high; it is simply not a meaningful statistic.

Histograms make correct aggregation possible

A Prometheus histogram exposes cumulative counters per bucket (le label): how many observations were less than or equal to 0.1 s, 0.25 s, and so on. Counters from different pods can be summed, because a count of requests under 0.25 s on pod A plus the same count on pod B is the count for both. After summing buckets, histogram_quantile estimates the quantile of the combined distribution.

So the order of operations is always:

1. rate() on the bucket counters, per series.
2. sum by (le, ...) to combine instances, keeping le.
3. histogram_quantile() last.

Summaries cannot be aggregated

Prometheus summaries compute quantiles inside each process and expose them as gauges. Those quantiles cannot be combined across instances at all. If you need fleet-wide percentiles, use histograms (or native histograms), not summaries.

Who is affected

Any dashboard or alert that applies avg, max or avg_over_time to quantile values: per-pod histogram quantiles averaged across pods, quantiles from Prometheus summaries averaged across instances, and percentile metrics from APM tools aggregated in a spreadsheet. max of per-pod p99 is less dangerous, because it does not hide the slow pod, but it is still not the p99 of user requests.

Environment

Dashboard: fleet-wide quantile plus per-pod breakdown

Two queries, two questions. The headline p99 for users:

histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{service="search"}[5m]))
)

And a per-pod p99 panel, shown as separate lines, not averaged, to find outliers:

histogram_quantile(0.99,
  sum by (pod, le) (rate(http_request_duration_seconds_bucket{service="search"}[5m]))
)

In the incident, the second panel would have shown one line far above the rest immediately.

Alert on the fleet-wide value and on outliers

- alert: SearchLatencyHigh
  expr: |
    histogram_quantile(0.99,
      sum by (le) (rate(http_request_duration_seconds_bucket{service="search"}[5m]))
    ) > 0.8
  for: 5m
  labels: { severity: page }

- alert: SearchPodLatencyOutlier
  expr: |
    histogram_quantile(0.99, sum by (pod, le) (rate(http_request_duration_seconds_bucket{service="search"}[5m])))
      > 3 *
    scalar(histogram_quantile(0.5, sum by (le) (rate(http_request_duration_seconds_bucket{service="search"}[5m]))))
  for: 10m
  labels: { severity: ticket }

The outlier alert compares each pod's p99 against the fleet median, catching a single bad instance even when the overall p99 is still acceptable. scalar() turns the label-less fleet median into a number, so it can be compared with every per-pod series without label matching.

Even better: alert on the share of slow requests

Quantile estimates from buckets are approximations, interpolated within a bucket. For alerting, a ratio of requests above a threshold is exact and easier to reason about:

1 - (
  sum(rate(http_request_duration_seconds_bucket{service="search", le="0.8"}[5m]))
    /
  sum(rate(http_request_duration_seconds_count{service="search"}[5m]))
)

This is "the fraction of requests slower than 800 ms", which maps directly to an SLO such as "99% of searches complete within 800 ms". It requires a bucket boundary at the threshold, so we added 0.8 to the bucket list.

Alternatives and trade-offs

• Native histograms in recent Prometheus versions give much finer resolution without choosing buckets, and aggregate the same way.
• Tracing-based latency analysis shows individual slow requests, complementary to metrics.

Communicating the change

Switching the headline query made the p99 line on the dashboard jump overnight, which looked like a regression to people who had not followed the incident. We annotated the dashboard with the date and the reason, and kept the old query in a hidden panel for two weeks, so anyone comparing could see that the service had not changed, only the measurement.

How to test the fix

On staging, inject latency into one pod with a fault injection flag and compare the old and new queries:

kubectl exec -n search search-api-0 -- curl -s -XPOST localhost:8080/debug/faults -d '{"latency_ms":3000}'

The averaged query moved modestly; the fleet-wide query and the outlier panel showed the problem within one evaluation window.

Confirming in production

We back-tested the new alert against the incident data: it would have fired seven minutes after the disk started failing, instead of not at all. Two weeks later, the outlier alert caught a pod with a noisy neighbour before any customer noticed.

Regression guard

• A lint rule in our dashboards-as-code repository flags avg(histogram_quantile(...)), avg_over_time on quantile gauges, and histogram_quantile with sum by that drops le.
• Latency SLOs are defined as slow-request ratios, not quantile thresholds.

Security implications

None directly. Misleading latency metrics delay incident detection of every kind, including performance degradation caused by abusive traffic against one instance.

Prevention checklist

• Never average percentiles across instances, time windows or services.
• Sum histogram buckets by le first, apply histogram_quantile last.
• Show per-instance percentiles as separate lines to find outliers.
• Prefer slow-request ratios for alerts and SLOs.
• Use histograms, not summaries, for anything you need to aggregate.

References

• Prometheus, Histograms and summaries: https://prometheus.io/docs/practices/histograms/
• Prometheus, histogram_quantile(): https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile
• Prometheus, Native histograms: https://prometheus.io/docs/specs/native_histograms/
• Google SRE Workbook, Implementing SLOs: https://sre.google/workbook/implementing-slos/