After adding Istio to our EKS cluster, every rollout produced a burst of failures: startup jobs could not reach the database and calls returned 503 upstream connect error. The application started before its Envoy sidecar was ready. How we proved it and fixed it with native sidecars and holdApplicationUntilProxyStarts.
After adding one environment variable in values-prod.yaml, production pods lost six others defined in the chart's defaults, including the database host. Helm deep-merges maps but replaces lists entirely. How we spotted it with helm template diffs and restructured values so overrides merge.
EKS Auto Mode hands node provisioning, core add-ons, load balancing and block storage to AWS. We moved two clusters from managed node groups and self-managed Karpenter. What Auto Mode manages, what changes for operators, the limits that surprised us, and the migration steps that kept workloads running.
Helm 4 is the first major Helm release in six years. We moved forty charts and three deployment pipelines from Helm 3. Notes on what changed for us in practice: server-side apply and field ownership, the new kstatus-based waiting, the plugin system, CLI changes, and the test plan that caught problems before production.
Kubernetes can now change a running pod's CPU and memory requests and limits without recreating it. How the resize subresource works, what resizePolicy controls, which changes still restart a container, how it interacts with QoS classes and VPA, and where we use it in production.
We moved CI to self-hosted GitHub Actions runners on EKS with Actions Runner Controller. Every job that built an image failed: Cannot connect to the Docker daemon. Runner pods have no Docker by default. How dind and kubernetes container modes work, what each costs in security, and the node pool we gave them.
We went from one cluster to nine and our Argo CD setup stopped scaling: hundreds of hand-written Application manifests and one controller everyone depended on. How app of apps and ApplicationSet compare, hub-and-spoke versus an Argo CD per cluster, and the layout we settled on.
We rotated the cosign key used to sign container images, and within minutes Kyverno rejected every new pod in production: no matching signatures. The policy trusted only the old public key, and images signed before the rotation needed it too. How we recovered and the staged rotation we use now.
During a blue-green release with Argo Rollouts, a third of production requests hit the unpromoted preview version: our Ingress pointed at a hand-made Service whose app label matched pods of both versions. How Rollouts manages selectors, how we found the leak, and keeping preview private.