Our pricing service scaled from 3 to 10 replicas under load, but one pod stayed at 100% CPU while others idled. gRPC uses long-lived HTTP/2 connections, and a ClusterIP Service balances connections, not requests. How we found it and fixed it with client-side load balancing, a headless Service and MaxConnectionAge.
We rolled out a default-deny egress NetworkPolicy to a namespace and every pod immediately failed to resolve names, including other services in the same namespace. Policies select pods, DNS runs in kube-system, and the namespace selector needed the right label. The policy set we use now and how we test it.
A partial outage in one AWS availability zone took down our search API completely, although it ran three replicas on a three-zone cluster. The scheduler had placed all three in the same zone. How the default spreading works, why it was not enough, and the topologySpreadConstraints we now require.
A dashboard service account had a Role and RoleBinding for its namespace, yet every request failed with forbidden at the cluster scope, because the client listed pods across all namespaces. How Roles, ClusterRoles and bindings combine, how to debug with kubectl auth can-i, and a fix without cluster-wide read.
A new KEDA ScaledObject was meant to scale workers from zero on SQS queue depth. The queue reached 40,000 messages and the Deployment stayed at zero. The KEDA operator, not the worker, needed permission to read the queue. How we found it in the ScaledObject status and fixed it with EKS Pod Identity.
We moved twelve services from IRSA to EKS Pod Identity to simplify IAM. Three of them started failing with AccessDenied or fell back to the node role. Causes: a missing agent add-on on one node group, an old AWS SDK that did not know the new credential provider, and a trust policy without sts:TagSession.
After a node group replacement, a Redis StatefulSet pod stayed Pending with 1 node(s) had volume node affinity conflict. Its EBS volume lived in eu-west-1a, and no node in that zone had room. How zonal volumes, WaitForFirstConsumer and topology interact, and how we made the StatefulSet survive node changes.
After upgrading to Karpenter v1 with WhenEmptyOrUnderutilized consolidation, our EKS cluster replaced nodes constantly and long-running pods were evicted every few minutes. How we read Karpenter's events, tuned consolidateAfter and disruption budgets, and protected jobs that must not be interrupted.
About one request in two hundred from our pods took exactly five seconds longer than normal. The delay was in DNS: parallel A and AAAA lookups over UDP raced in the kernel's conntrack table and one reply was dropped, so glibc waited its 5-second timeout. How we proved it and fixed it with NodeLocal DNSCache.