After adding Istio to our EKS cluster, every rollout produced a burst of failures: startup jobs could not reach the database and calls returned 503 upstream connect error. The application started before its Envoy sidecar was ready. How we proved it and fixed it with native sidecars and holdApplicationUntilProxyStarts.
Our pricing service scaled from 3 to 10 replicas under load, but one pod stayed at 100% CPU while others idled. gRPC uses long-lived HTTP/2 connections, and a ClusterIP Service balances connections, not requests. How we found it and fixed it with client-side load balancing, a headless Service and MaxConnectionAge.
New Pods failed with CreateContainerConfigError while old ones kept running. A key renamed in the cloud secret store made External Secrets rewrite the Kubernetes Secret without DATABASE_URL. We show how to find the missing key fast and how to fail before rollout next time.
We rolled out a default-deny egress NetworkPolicy to a namespace and every pod immediately failed to resolve names, including other services in the same namespace. Policies select pods, DNS runs in kube-system, and the namespace selector needed the right label. The policy set we use now and how we test it.
After adding one environment variable in values-prod.yaml, production pods lost six others defined in the chart's defaults, including the database host. Helm deep-merges maps but replaces lists entirely. How we spotted it with helm template diffs and restructured values so overrides merge.
EKS Auto Mode hands node provisioning, core add-ons, load balancing and block storage to AWS. We moved two clusters from managed node groups and self-managed Karpenter. What Auto Mode manages, what changes for operators, the limits that surprised us, and the migration steps that kept workloads running.
Passing --workers 4 to docker run crashed the container with exec: "--workers": executable file not found. Arguments after the image name replace CMD entirely; they do not append. We explain the override rules in Docker, Compose and Kubernetes and the image layout that makes extra flags safe.
Terraform created our new EKS cluster cleanly, then nobody could use it: kubectl said we must be logged in, and the Helm step in CI failed the same way. The cluster had API-only authentication, no creator admin, and access entries for the wrong role ARNs.
Our vLLM deployment had an HPA targeting 70 percent CPU. Under a load spike, requests queued for 40 seconds while CPU sat at 15 percent, so nothing scaled. GPU-bound servers need to scale on queue depth. We switched to KEDA on vLLM's waiting-requests metric and fixed warm-up.
Helm 4 is the first major Helm release in six years. We moved forty charts and three deployment pipelines from Helm 3. Notes on what changed for us in practice: server-side apply and field ownership, the new kstatus-based waiting, the plugin system, CLI changes, and the test plan that caught problems before production.
Our nightly export finished its work in 20 minutes, yet the Job sat at 0/1 completions for six hours and then failed with DeadlineExceeded. The database proxy sidecar never exited. We replaced shutdown hacks with native sidecars (restartPolicy: Always init containers) and show a full repro.
Our managed node group upgrade stalled for two hours on one node. A PodDisruptionBudget with minAvailable equal to the replica count meant no pod could ever be evicted. We switched to maxUnavailable, fixed replica counts and added a pre-upgrade PDB check.
Kubernetes can now change a running pod's CPU and memory requests and limits without recreating it. How the resize subresource works, what resizePolicy controls, which changes still restart a container, how it interacts with QoS classes and VPA, and where we use it in production.
We moved CI to self-hosted GitHub Actions runners on EKS with Actions Runner Controller. Every job that built an image failed: Cannot connect to the Docker daemon. Runner pods have no Docker by default. How dind and kubernetes container modes work, what each costs in security, and the node pool we gave them.
We went from one cluster to nine and our Argo CD setup stopped scaling: hundreds of hand-written Application manifests and one controller everyone depended on. How app of apps and ApplicationSet compare, hub-and-spoke versus an Argo CD per cluster, and the layout we settled on.
Calls from our pods to a payment API had 40 to 200 ms of unexplained latency and CoreDNS was the busiest workload in the cluster. Every lookup of api.payments.example went through five search domains first because of ndots:5. FQDNs with a trailing dot and a lower ndots fixed both.
A traffic spike scaled our API from 8 to 30 pods, each with a 20-connection pool, and Postgres started refusing connections. Raising max_connections would have made it worse. We put PgBouncer in transaction mode in front and sized pools from the database side.
A partial outage in one AWS availability zone took down our search API completely, although it ran three replicas on a three-zone cluster. The scheduler had placed all three in the same zone. How the default spreading works, why it was not enough, and the topologySpreadConstraints we now require.
Our nightly invoice export stopped running and nobody noticed for nine days. One Job had hung on a network call with no timeout, and with concurrencyPolicy Forbid every later schedule was skipped. Deadlines on the Job, timeouts in code and an alert on last success fixed it.
Our queue workers took exactly 30 seconds to stop on every deploy and lost the job they were processing. Exit code 137 in every pod. The Dockerfile used shell-form CMD, so /bin/sh was PID 1 and never forwarded SIGTERM to Node. Exec form plus a real shutdown handler fixed it.
During a traffic spike, new GPU pods for our LLM service took nine minutes to become ready, and users waited or got errors. The image contained 15 GB of model weights, so every new node pulled the whole thing. Separating weights from the image and caching them per node brought scale-up under two minutes.
Kubernetes announced the retirement of ingress-nginx, with best-effort maintenance ending in March 2026. This is the plan we used to inventory annotations, map them to Gateway API, run both side by side and cut over host by host without downtime.
Our GPU node pool stayed at eight nodes from Friday night to Monday morning with almost no GPU usage. The cluster autoscaler refused to remove them: notebook pods with local storage, a bare debug pod and a kube-system pod without a PDB. Fixing each blocker cut GPU spend by 38 percent.
A dashboard service account had a Role and RoleBinding for its namespace, yet every request failed with forbidden at the cluster scope, because the client listed pods across all namespaces. How Roles, ClusterRoles and bindings combine, how to debug with kubectl auth can-i, and a fix without cluster-wide read.
Both KServe and KubeRay can put vLLM behind an OpenAI-compatible endpoint on Kubernetes. They differ in who owns the control plane, how scaling decisions are made and how multi-node models are handled. We ran both for the same model and this is how we chose.