A canary that a human watches on a dashboard is only as good as that human's attention. This guide covers automated canary analysis: which metrics to gate on, how to size steps and durations, how to handle low traffic, and the traps that make canaries pass when they should fail.
Our vLLM deployment had an HPA targeting 70 percent CPU. Under a load spike, requests queued for 40 seconds while CPU sat at 15 percent, so nothing scaled. GPU-bound servers need to scale on queue depth. We switched to KEDA on vLLM's waiting-requests metric and fixed warm-up.
During a partial outage, our latency dashboard showed p99 at 350 ms while one pod in twelve was taking seconds per request. The panel averaged per-pod p99 values. Percentiles cannot be averaged. How we fixed the query by aggregating histogram buckets first, and what that changed for alerts and SLOs.
A developer added user_id as a label on a request latency histogram to debug slow accounts. Active time series went from 1.2 million to 19 million, our hosted metrics bill quadrupled and the query layer slowed down. We found the culprit, dropped the label and set cardinality limits.
We shipped a ServiceMonitor for a new service and its metrics never appeared: no target, no error, nothing in the logs. Prometheus Operator only picks ServiceMonitors that match the Prometheus resource's selectors, and our chart required a release label. Then a port name bit us too.
After we changed the scrape interval from 15 to 60 seconds to save money, our request-rate panels went blank in places and alerts flapped. rate() needs at least two samples in its range, and our queries used [1m]. Why the range must be several scrape intervals, how $__rate_interval works, and how we fixed alerts too.