Grafana panel empty or spiky after a scrape interval change: rate() range too short

To reduce the cost of our managed Prometheus-compatible metrics service, we changed the scrape interval for most application targets from 15 seconds to 60 seconds. Ingested samples dropped by 75%, as planned. The next morning, three things were wrong:

• Request-rate panels on several dashboards were blank for some time ranges, and spiky with gaps for others.
• An alert on error rate fired and resolved repeatedly overnight.
• Some recording rules produced no data at all.

The queries all looked like this:

sum by (service) (rate(http_requests_total{env="prod"}[1m]))

Testing the query directly

In the Prometheus expression browser, for one service, with a range of 1 minute:

rate(http_requests_total{service="orders-api", env="prod"}[1m])

The result alternated between a value and no value, depending on the evaluation timestamp. With [5m]:

rate(http_requests_total{service="orders-api", env="prod"}[5m])

A smooth, continuous line.

Looking at the raw samples explained it:

http_requests_total{service="orders-api", env="prod", instance="10.20.31.7:8080"}[2m]

8312004 @1748937600.123
8312398 @1748937660.119

Two samples in two minutes, one every 60 seconds. A one-minute window contains at most one sample, and sometimes, depending on alignment and scrape jitter, none.

What we ruled out

• Missing targets. up was 1 for every target, continuously.
• Counter resets. resets() over the period was zero for most series; resets cause dips, not gaps.
• Grafana caching. Queries in the Prometheus UI showed the same gaps.

Root cause

rate() calculates the per-second increase from the samples inside its range window and needs at least two samples to produce a value. Our queries used a one-minute window. With a 60-second scrape interval, a one-minute window often contains a single sample, so rate() returns nothing for that evaluation, and when it does find two samples, extrapolation over such a short window produces noisy values.

How rate() works

For each series, rate(counter[range]) takes the samples within the range, computes the increase between the first and last (accounting for counter resets), extrapolates to the window edges, and divides by the window length. With fewer than two samples, there is no increase to compute.

The usual rule of thumb: the range should be at least four times the scrape interval. That leaves room for a missed scrape and for scrape timing jitter while still having several samples.

Grafana's $__rate_interval

Grafana's $__interval is the time between points on the graph, which depends on the dashboard time range and panel width. Using [$__interval] with rate() breaks when zoomed in: the interval can drop below the scrape interval. Grafana provides $__rate_interval for exactly this case. It is calculated as the maximum of $__interval plus one scrape interval and four times the scrape interval, using the scrape interval configured on the Prometheus data source.

That last part matters: if the data source still says 15 seconds while targets are scraped every 60, $__rate_interval is calculated from the wrong value.

rate versus irate

irate() uses only the last two samples in the range. It reacts quickly but is noisy and still needs two samples. It is meant for fast-moving graphs, not for alerts or recording rules.

Who is affected

Anyone who increases scrape intervals to save cost or load, anyone whose dashboards were built with [1m] habits from a 15-second setup, and teams using [$__interval] instead of [$__rate_interval], which breaks when zooming in. Federated or downsampled data, where effective resolution is lower than the original scrape interval, has the same problem with short ranges. increase() is affected in the same way, since it is rate() multiplied by the range.

Environment

Dashboards: use $__rate_interval and set the data source interval

First, the Prometheus data source in Grafana got the correct scrape interval:

# provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: https://prometheus.internal.example.com
    jsonData:
      timeInterval: 60s

Then panel queries changed from hard-coded ranges to the variable:

- sum by (service) (rate(http_requests_total{env="prod"}[1m]))
+ sum by (service) (rate(http_requests_total{env="prod"}[$__rate_interval]))

We did it with a script over the dashboard JSON files in our repository, since all dashboards are provisioned from code.

Alerts and recording rules: fixed, sufficient ranges

Alert rules and recording rules are evaluated by Prometheus, not Grafana, so $__rate_interval does not exist there. They need explicit ranges sized from the scrape interval:

groups:
  - name: orders-api
    rules:
      - record: service:http_requests:rate5m
        expr: sum by (service) (rate(http_requests_total{env="prod"}[5m]))

      - alert: HighErrorRate
        expr: |
          sum by (service) (rate(http_requests_total{env="prod", status=~"5.."}[5m]))
            /
          sum by (service) (rate(http_requests_total{env="prod"}[5m]))
          > 0.02
        for: 10m
        labels:
          severity: page

The for: 10m clause also stops a single noisy evaluation from paging anyone. The recording rule's name follows the convention of including the range (rate5m), so it is obvious later which window was used.

Keep a faster scrape where it matters

A few targets need fine-grained data: the load balancer exporter and the payment service's latency histograms used for SLO alerts. Those kept a 15-second interval through a separate scrape job. Everything else stayed at 60 seconds.

Alternatives and trade-offs

• Revert to 15-second scrapes. Simplest, at four times the cost.
• Recording rules at the old resolution. Pre-aggregating with recording rules reduces query cost but not ingestion cost, which was the problem.

How to test the fix

count_over_time(http_requests_total{service="orders-api", env="prod"}[5m])

Every series should return at least 4 samples. For alert rules, promtool test rules with a fixture at 60-second spacing confirms the rule fires as expected:

tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{service="orders-api",env="prod",status="500"}'
        values: '0+30x30'
      - series: 'http_requests_total{service="orders-api",env="prod",status="200"}'
        values: '0+600x30'
    alert_rule_test:
      - eval_time: 20m
        alertname: HighErrorRate
        exp_alerts:
          - exp_labels: { service: orders-api, severity: page }

Confirming in production

All request-rate panels showed continuous lines at every zoom level. The error-rate alert stopped flapping; over the next month it fired twice, both real incidents. The recording rules produced data again, and the SLO burn-rate dashboards built on them recovered.

Regression guard

• A CI check rejects alert and recording rules whose rate(), increase() or irate() range is less than four times the target scrape interval.
• A dashboard lint rejects hard-coded ranges in rate() in favour of $__rate_interval.
• Scrape interval changes now require a search for affected queries in the same pull request.

Security implications

None directly. Flapping alerts are an operational risk: teams start ignoring them, which delays response to real incidents, including security-relevant ones such as sudden error spikes from attacks.

Prevention checklist

• Size rate() ranges at four times the scrape interval or more.
• Use $__rate_interval in Grafana panels, and set the data source scrape interval correctly.
• Use fixed, explicit ranges in alerts and recording rules, and name recording rules after them.
• Add a for duration to alerts based on rates.
• Review queries whenever scrape intervals change.

References

• Prometheus, rate() function: https://prometheus.io/docs/prometheus/latest/querying/functions/#rate
• Grafana, Prometheus template variables and $__rate_interval: https://grafana.com/docs/grafana/latest/datasources/prometheus/template-variables/
• Prometheus, Recording rules naming: https://prometheus.io/docs/practices/rules/
• promtool unit testing for rules: https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/