#cardinality

DevOps & CI/CD Bug Report

Monitoring bill jumped after one label change: high-cardinality custom metrics

Ryan Pham · Mar 18, 2026 · 5 min read

A developer added user_id as a label on a request latency histogram to debug slow accounts. Active time series went from 1.2 million to 19 million, our hosted metrics bill quadrupled and the query layer slowed down. We found the culprit, dropped the label and set cardinality limits.

DevOps & CI/CD Bug Report

Loki rejected our logs with "maximum active stream limit exceeded": high-cardinality labels

Elena Kovač · Mar 4, 2026 · 7 min read

A new release started dropping logs in Loki with a 429 and stream limit errors. A logging change had promoted user_id and trace_id to labels, creating hundreds of thousands of streams. How Loki's streams work, how we found the offending labels, and how structured metadata and query-time parsing replaced them.