After a node group replacement, a Redis StatefulSet pod stayed Pending with 1 node(s) had volume node affinity conflict. Its EBS volume lived in eu-west-1a, and no node in that zone had room. How zonal volumes, WaitForFirstConsumer and topology interact, and how we made the StatefulSet survive node changes.
We ran everything in one big Kubernetes cluster until a CRD upgrade broke three teams at once. This guide compares one shared cluster, a cluster per environment and a cluster per team on isolation, blast radius, upgrade effort and real cost, with a worked cost model.
After upgrading to Karpenter v1 with WhenEmptyOrUnderutilized consolidation, our EKS cluster replaced nodes constantly and long-running pods were evicted every few minutes. How we read Karpenter's events, tuned consolidateAfter and disruption budgets, and protected jobs that must not be interrupted.
Every deploy of our Go pricing API produced 30 to 60 seconds of 502 and 503 errors. New pods got traffic before they were ready, and old pods exited while the ingress still routed to them. A readiness probe, a preStop sleep and real graceful shutdown made rollouts clean.
Our self-managed cluster pulled images from ECR fine after setup and failed every morning with no basic auth credentials. The imagePullSecret held an ECR token that expires after 12 hours. We switched to the kubelet credential provider so nodes fetch fresh tokens themselves.
Every deploy of one service failed with another operation is in progress after someone cancelled a pipeline mid-upgrade. The release was stuck in pending-upgrade. We recovered with helm rollback, then added CI concurrency, timeouts and atomic upgrades so it cannot recur.
Training jobs that ran fine on a workstation crashed in containers with a bus error in DataLoader workers. Workers pass batches through shared memory, and containers get only 64 MB of /dev/shm by default. A memory-backed emptyDir in Kubernetes and --shm-size in Docker fixed it.
During peak traffic a third of our traces arrived with missing spans, and the ones we needed for incident review were exactly the ones gone. The Collector was refusing data at its memory limit and dropping from a full export queue. Here is how we measured the loss and sized the pipeline.
A release added a column and our new Pods went Ready and served traffic before the migration Job had even connected to the database. Argo CD applies everything in one wave together. We moved the migration into a PreSync hook, ordered dependencies with sync waves, and show a runnable repro.
Our storefront BFF pods restarted every few hours with OOMKilled and exit code 137, then fell into CrashLoopBackOff during the holiday peak. A max-old-space-size inherited from the VM days was double the container memory limit. Sizing the heap to the limit and bounding a cache fixed it.
About one request in two hundred from our pods took exactly five seconds longer than normal. The delay was in DNS: parallel A and AAAA lookups over UDP raced in the kernel's conntrack table and one reply was dropped, so glibc waited its 5-second timeout. How we proved it and fixed it with NodeLocal DNSCache.
Half our Argo CD applications showed OutOfSync seconds after every successful sync, and auto-sync kept fighting the HPA over replica counts. The live objects had fields set by controllers and webhooks that Git never mentioned. Targeted ignoreDifferences and server-side diff fixed it.
A routine rolling deploy restarted one realtime pod holding 50,000 WebSocket connections. Every client reconnected within two seconds, the auth and history lookups melted, and the storm kept itself alive for 25 minutes. Here is how we drain sockets, pace reconnects and make resume cheap.
kubectl apply returns in milliseconds, but a container can take a minute to start. We trace every hop: API server, admission, etcd, controllers, scheduler, kubelet, CRI, CNI and probes, and show which command reveals each stage.