Loki rejected our logs with "maximum active stream limit exceeded": high-cardinality labels

On a Wednesday morning, the on-call engineer opened Grafana to investigate a support ticket and found no logs for the payments service after 08:40. The service was running and serving traffic. The log agent's own logs were full of errors:

level=warn component=loki.write msg="error sending batch, will retry" status=429
error="server returned HTTP status 429 Too Many Requests (429): Maximum active stream limit exceeded
when trying to create stream {app=\"payments-api\", env=\"prod\", ...}, reduce the number of active streams
(reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be
increased, user: 'prod'"

The agent retried and eventually dropped batches. A release of the payments service had gone out at 08:35.

What changed in the release

The release included a "logging improvement": the service's structured logger now added user_id, trace_id and request_path as fields on every log line, and the agent configuration, shared through the same pull request, promoted those fields to Loki labels so they would be "easy to filter on".

// Grafana Alloy pipeline, the problematic part
loki.process "payments" {
  stage.json {
    expressions = { level = "level", user_id = "user_id", trace_id = "trace_id", request_path = "path" }
  }
  stage.labels {
    values = { level = "", user_id = "", trace_id = "", request_path = "" }
  }
  forward_to = [loki.write.default.receiver]
}

Measuring the streams

Loki's series API counts streams matching a selector:

logcli series '{app="payments-api", env="prod"}' --since=30m --analyze-labels

Total Streams:  412,883
Unique Labels:  7

Label Name     Unique Values  Found In Streams
trace_id       401,210        412,883
user_id         38,447        412,883
request_path     1,962        412,883
pod                 12        412,883
level                4        412,883
app                  1        412,883
env                  1        412,883

Before the release, the service had about 48 streams (12 pods times 4 levels). Afterwards, over 400,000 in thirty minutes, mostly one stream per trace.

What we ruled out

• An ingestion rate limit. The error explicitly named the stream limit, not the bytes-per-second limit.
• A Loki outage. Other services' logs were ingested normally.
• Log volume. The number of log lines had barely changed; only their labels had.

Root cause

Every unique combination of label values in Loki is a separate stream. Promoting unbounded values such as trace_id, user_id and request paths with IDs in them to labels turned a handful of streams into hundreds of thousands. The tenant hit its per-tenant active stream limit, and Loki rejected new streams, so the payments service's logs were dropped.

Labels are an index, not a filter

Loki indexes labels only. Each unique label set is a stream, stored as compressed chunks. Queries first select streams by labels, then scan the chunks of those streams for content. This design is what makes Loki cheap: a small index and large, compressed chunks.

High-cardinality labels break that model:

• Each stream has overhead in the ingesters (memory) and in the index.
• Many tiny streams produce many tiny chunks, which compress and query poorly.
• Per-tenant limits on active streams protect the cluster, and new streams get rejected.

What should be a label

The rule: a label should describe where the log comes from, not what it says.

Structured metadata

Loki 3.x supports structured metadata: key-value pairs attached to each log line that are stored with the line, not in the index. They do not create streams, and they can be filtered on at query time without parsing the line. It is designed exactly for values like trace ids.

Who is affected

Teams that promote parsed JSON fields to labels for convenience, Kubernetes setups that turn every pod label or annotation into a Loki label, and pipelines that label by HTTP path, status code with request ids, or customer id. Single-tenant setups without stream limits do not reject logs, but they degrade instead: ingesters use more memory and queries over many tiny chunks become slow.

Environment

Immediate: stop creating streams

We reverted the agent configuration change first, which restored ingestion within minutes. Existing high-cardinality streams went inactive after the idle period and stopped counting against the limit.

Keep the fields, as structured metadata

The goal of the change, filtering logs by trace and user, was reasonable. We kept it with structured metadata instead of labels:

loki.process "payments" {
  stage.json {
    expressions = { level = "level", user_id = "user_id", trace_id = "trace_id" }
  }
  stage.labels {
    values = { level = "" }
  }
  stage.structured_metadata {
    values = { trace_id = "", user_id = "" }
  }
  forward_to = [loki.write.default.receiver]
}

Queries filter on structured metadata after selecting streams by labels:

{app="payments-api", env="prod"} | trace_id="4bf92f3577b34da6a3ce929d0e0e4736"

{app="payments-api", env="prod", level="error"} | user_id="u_48121"

The stream selector keeps the search to the payments streams; the filter then matches on the metadata field without parsing the JSON body.

Parse at query time when needed

For fields not in structured metadata, LogQL parsers extract them during the query:

sum by (path) (
  count_over_time({app="payments-api", env="prod"} | json | __error__="" | status >= 500 [5m])
)

Query-time extraction costs CPU at query time, but only for the streams and time range being queried.

Trace-to-logs links

Grafana's tracing data source can link a span to logs using a query template on trace_id. With structured metadata, the generated query is fast, which delivers the "easy filtering" the original change wanted.

Alternatives and trade-offs

• Raising the stream limit. Temporary relief at higher memory cost; the next unbounded label breaks it again.
• Splitting the tenant. Moves the problem rather than fixing it.

How to test the fix

logcli series '{app="payments-api", env="prod"}' --since=1h --analyze-labels | head -3
# Total Streams: 48
logcli query '{app="payments-api", env="prod"} | trace_id="4bf92f3577b34da6a3ce929d0e0e4736"' --since=1h --limit 5

Confirming in production

Active streams for the payments tenant returned to their usual level of a few thousand across all services. Searches by trace_id over the last hour returned in under two seconds. The agent's retry and drop counters went to zero.

Regression guard

• A CI check on agent configuration fails if any stage.labels block contains a key outside an allow-list (app, namespace, env, cluster, level, pod, container).
• An alert on each tenant's active streams above 70% of its limit.
• An alert on the agent's dropped entries metric.

Security implications

Labels in Loki are also visible in label browsers and autocomplete for everyone with read access to the tenant. Promoting user_id to a label exposed a list of active user ids in the Grafana label picker. Structured metadata does not appear there, and the log pipeline now redacts email addresses and tokens before shipping.

Prevention checklist

• Use labels only for the source of logs: app, namespace, environment, level.
• Put trace ids, user ids and request ids in structured metadata.
• Parse other fields at query time.
• Allow-list labels in agent configuration and review changes to it.
• Alert on active streams per tenant and on dropped log entries.

References

• Grafana Loki, Label best practices: https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/
• Grafana Loki, Structured metadata: https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/
• Grafana Loki, Cardinality: https://grafana.com/docs/loki/latest/get-started/labels/cardinality/
• LogQL log queries: https://grafana.com/docs/loki/latest/query/log_queries/