A developer added user_id as a label on a request latency histogram to debug slow accounts. Active time series went from 1.2 million to 19 million, our hosted metrics bill quadrupled and the query layer slowed down. We found the culprit, dropped the label and set cardinality limits.
A cloud bill that grows 30 percent can be great news or a disaster, depending on what grew with it. Unit costs, such as cost per active customer or per thousand requests, tell you which. This guide shows how to choose units, allocate shared costs and use the numbers in decisions.
After we changed the scrape interval from 15 to 60 seconds to save money, our request-rate panels went blank in places and alerts flapped. rate() needs at least two samples in its range, and our queries used [1m]. Why the range must be several scrape intervals, how $__rate_interval works, and how we fixed alerts too.