Tag See all →

#grpc

2 stories

Networking & DNS Bug Report

Scaled to 10 pods, one pod at 100% CPU: gRPC connections pinned behind a ClusterIP Service

Priya Raghavan · Sep 29, 2026 · 7 min read

Our pricing service scaled from 3 to 10 replicas under load, but one pod stayed at 100% CPU while others idled. gRPC uses long-lived HTTP/2 connections, and a ClusterIP Service balances connections, not requests. How we found it and fixed it with client-side load balancing, a headless Service and MaxConnectionAge.

Banner: a telemetry pipeline overflowing while trace spans fall out of a full queue

DevOps & CI/CD Bug Report

OpenTelemetry Collector dropping spans under load: memory_limiter, batch and queue tuning

John Nguyen · Dec 28, 2024 · 10 min read

During peak traffic a third of our traces arrived with missing spans, and the ones we needed for incident review were exactly the ones gone. The Collector was refusing data at its memory limit and dropping from a full export queue. Here is how we measured the loss and sized the pipeline.