Scaled to 10 pods, one pod at 100% CPU: gRPC connections pinned behind a ClusterIP Service

During a sale, the pricing service, which the checkout service calls over gRPC for every cart, ran hot. The HorizontalPodAutoscaler did its job and scaled from 3 to 10 replicas. Latency did not improve. The CPU graph per pod told the story:

pricing-6c9f7b-2xk4p   CPU 1980m / 2000m
pricing-6c9f7b-8vq2n   CPU   95m
pricing-6c9f7b-b7tlm   CPU  110m
pricing-6c9f7b-h2wxq   CPU   88m
... (six more between 60m and 120m)

One pod did nearly all the work. The new replicas received almost nothing.

Connections, not requests

The pricing pods exposed gRPC server metrics. Request rate per pod matched the CPU picture. The question was why traffic did not spread. On the busy pod, open connections from clients:

kubectl exec -n pricing pricing-6c9f7b-2xk4p -- ss -tnH state established '( sport = :50051 )' | wc -l
# 4

Four TCP connections, one from each of the four checkout pods. On the new pods: zero or one. Each checkout pod had opened a single HTTP/2 connection to pricing.pricing.svc.cluster.local:50051 at startup, hours earlier, when only three pricing pods existed, and had used that connection for every call since.

What we ruled out

• Uneven request cost. The same cart sizes went to every pod in a test with fresh connections.
• Readiness. All ten pods were Ready and listed in the Service endpoints.
• kube-proxy mode. iptables and IPVS behave the same here: both balance at connection time.

Root cause

A ClusterIP Service load-balances TCP connections. gRPC multiplexes all requests from a client over one long-lived HTTP/2 connection. Once a connection is established to a pod, every request on it goes to that pod. New replicas only receive traffic from new connections, which our clients rarely made. Three of the original pods had absorbed most of the original connections; after a checkout deploy, one pod happened to get all four.

Layer 4 balancing meets a multiplexed protocol

kube-proxy chooses a backend when a connection is created and does not look at individual requests. That is fine for many protocols and bad for gRPC.

Why the imbalance got worse over time

Connections only rebalance when they are re-established. Our checkout pods restarted with each deploy, roughly daily, and opened one connection each at startup. Pricing pods added by the HPA during a peak received connections only from checkout pods that started after them. Scaling down then removed the least busy pricing pods first, which were the new ones, and the cycle repeated. The cluster was spending money on replicas that were effectively idle, while users waited on the one pod that held the connections. The HPA saw average CPU well above target and kept scaling, without improving anything.

Who is affected

Any gRPC or HTTP/2 client calling a service through a ClusterIP Service or a layer 4 load balancer, including network load balancers: Java, Go, Python and Node.js gRPC clients all use one connection per target by default. HTTP/1.1 clients with small keep-alive pools are affected to a lesser degree.

Environment

There are three approaches; we combined two.

Option 1: client-side load balancing with a headless Service

A headless Service (clusterIP: None) makes DNS return the individual pod IPs instead of a single virtual IP. gRPC clients can then open a connection to each pod and spread requests with the round_robin policy.

apiVersion: v1
kind: Service
metadata:
  name: pricing-grpc
  namespace: pricing
spec:
  clusterIP: None
  selector:
    app: pricing
  ports:
    - name: grpc
      port: 50051

ManagedChannel channel = ManagedChannelBuilder
    .forTarget("dns:///pricing-grpc.pricing.svc.cluster.local:50051")
    .defaultLoadBalancingPolicy("round_robin")
    .usePlaintext()
    .build();

The dns:/// scheme tells grpc-java to resolve all addresses. One catch: the client only re-resolves DNS when connections fail or on its own schedule, so newly added pods may not be discovered for a while. That is what option 2 addresses.

Option 2: MaxConnectionAge on the server

The server can close connections after a maximum age, gracefully, so clients reconnect and re-resolve:

srv := grpc.NewServer(
    grpc.KeepaliveParams(keepalive.ServerParameters{
        MaxConnectionAge:      2 * time.Minute,
        MaxConnectionAgeGrace: 30 * time.Second,
    }),
)

Before closing, the server sends an HTTP/2 GOAWAY; clients finish in-flight calls and open a new connection, which with a ClusterIP Service may land on any pod, and with a headless Service triggers a new DNS resolution. This alone already fixes the worst imbalance, even without client changes.

Option 3: a proxy that understands HTTP/2

A service mesh sidecar (Envoy-based meshes, Linkerd) or an L7 proxy balances individual gRPC requests across endpoints. If you already run a mesh, it is the cleanest option, since no application code changes. We did not run a mesh and did not want to add one for this.

What we deployed

Server-side MaxConnectionAge of two minutes, and the checkout clients moved to the headless Service with round_robin. We kept the ClusterIP Service for other consumers that had not migrated yet.

Choosing between them

For a handful of internal gRPC services, the first two together are enough. Beyond that, the operational cost of a mesh starts to pay for itself.

How to test the fix

# generate load, then compare per-pod request rates
ghz --insecure --call pricing.v1.Pricing/Quote -d '{"cart_id":"c_123"}' -c 50 -z 2m \
  pricing-grpc.pricing.svc.cluster.local:50051
kubectl top pods -n pricing

Confirming in production

At the next traffic peak, the ten pricing pods showed CPU within 15% of each other, p99 latency dropped from 1.8 s to 140 ms, and the HPA settled at six replicas instead of hitting its maximum, because the capacity was finally used.

Regression guard

• A dashboard panel shows the coefficient of variation of request rate across pods for each gRPC service, with an alert when one pod takes more than twice the average.
• A shared gRPC server library sets MaxConnectionAge by default.
• Client libraries default to round_robin with headless Services for internal gRPC calls.

Security implications

None directly. Note that headless Services expose pod IPs to clients, which matters if you rely on NetworkPolicy by Service: policies are enforced on pod IPs anyway, so nothing changes, but client-side balancing means clients connect to pods directly, and mTLS or authorization must be enforced by the pods themselves.

Prevention checklist

• Do not expect a ClusterIP Service to balance gRPC or other HTTP/2 traffic per request.
• Use client-side round_robin with a headless Service, or an L7 proxy or mesh.
• Set MaxConnectionAge on gRPC servers so connections rebalance after scaling.
• Monitor request distribution per pod, not just totals.
• Load test after scaling events, not only at steady state.

References

• gRPC, Load balancing: https://grpc.io/blog/grpc-load-balancing/
• Kubernetes, Headless Services: https://kubernetes.io/docs/concepts/services-networking/service/#headless-services
• grpc-go keepalive and MaxConnectionAge: https://pkg.go.dev/google.golang.org/grpc/keepalive
• gRPC Java name resolution and load balancing: https://grpc.io/docs/guides/custom-load-balancing/