DNS, TLS/SSL, proxies, load balancers, CDN, timeouts.
After adding Istio to our EKS cluster, every rollout produced a burst of failures: startup jobs could not reach the database and calls returned 503 upstream connect error. The application started before its Envoy sidecar was ready. How we proved it and fixed it with native sidecars and holdApplicationUntilProxyStarts.
Our primary region could not write to its database for 38 minutes, yet Route 53 kept answering with the primary endpoint. The health check was green the whole time. Then automatic failback broke writes a second time. Here is why, and the health check design that fixed both.
Our pricing service scaled from 3 to 10 replicas under load, but one pod stayed at 100% CPU while others idled. gRPC uses long-lived HTTP/2 connections, and a ClusterIP Service balances connections, not requests. How we found it and fixed it with client-side load balancing, a headless Service and MaxConnectionAge.
We rolled out a default-deny egress NetworkPolicy to a namespace and every pod immediately failed to resolve names, including other services in the same namespace. Policies select pods, DNS runs in kube-system, and the namespace selector needed the right label. The policy set we use now and how we test it.
Our first Site-to-Site VPN between an office firewall and AWS came up green on both sides, yet pings failed and, once they worked, large file transfers hung. Three separate problems: route propagation, the firewall's traffic selectors and MTU. How we found each one.
Part 7 of the security fundamentals series. Firewalls, intrusion detection and prevention, VPNs and forward and reverse proxies compared by what they can see, what they can stop, where they sit, and the misconfigurations we find most often.
After a spot node reclaim, our Spring Cloud Gateway sent traffic to instances that no longer existed for over six minutes. Eureka had entered self-preservation and stopped evicting, while three layers of caches added their own delay. Here is how we cut stale routing to seconds.
After a framework upgrade, our CDN cache hit ratio for static assets dropped from 97 to 41 percent and origin data transfer doubled. The new session middleware attached Set-Cookie and Cache-Control private to every response, including JavaScript bundles. Scoping middleware fixed it.
Part 2 of the security fundamentals series. The networking concepts you need to reason about exposure: IP and MAC addresses, ports, TCP versus UDP, common services, and how to see what your own hosts are listening on.
Instances in a freshly added private subnet timed out on dnf and never showed up in Session Manager, while identical instances in older subnets were fine. The subnet had silently fallen back to the VPC main route table, which had no NAT route.
Every night API latency tripled for an hour and the 1 Gbit uplink of one server sat at 100%. We traced it from interface counters to a host, a socket and finally a backup job with no rate limit, then shaped it with tc and fixed the schedule.
Calls from our pods to a payment API had 40 to 200 ms of unexplained latency and CoreDNS was the busiest workload in the cluster. Every lookup of api.payments.example went through five search domains first because of ndots:5. FQDNs with a trailing dot and a lower ndots fixed both.
Kubernetes announced the retirement of ingress-nginx, with best-effort maintenance ending in March 2026. This is the plan we used to inventory annotations, map them to Gateway API, run both side by side and cut over host by host without downtime.
We moved a dozen small services from Nginx to Caddy and kept Nginx in front of two others. This is the honest comparison: automatic HTTPS, config side by side, what performance looked like, and the migration traps around headers, body limits and certificate storage.
A summary of Cloudflare's official postmortem for the November 18, 2025 outage: a database permission change doubled a generated feature file, which crossed a hard limit and crashed the proxy. Plus the lessons we are applying to our own config pipelines.
When a service cannot reach an API, guessing wastes hours. We follow the packet: name resolution, route, TCP handshake, TLS, then HTTP, with one or two commands per layer that each prove or eliminate a whole class of causes.
Most HTTP bugs can be pinned down with one tool you already have. This guide shows the curl options we use daily: verbose output, header-only requests, --resolve and --connect-to for testing servers before DNS changes, timing breakdowns, protocol versions and TLS checks.
A new subdomain's certificate sat in pending for hours with a self check error: wrong status code 404, expected 200. The challenge solver created its Ingress with an ingress class our controller ignored. Setting the class on the solver and checking the redirect path fixed issuance.
Firewall rules describe what you meant to expose; a scan from outside shows what you actually expose. This is the repeatable nmap routine we run against our own hosts, including the Docker port that slipped past ufw and how we diff results week to week.
After moving a service to a new Helm chart, every call to it failed with Connection refused while the Pods were Running and Ready. The Service selector no longer matched the Pod labels, and once that was fixed a wrong targetPort kept refusing connections. Here is the debugging path.
A low but stubborn rate of 502s at the load balancer and socket hang up errors between services turned out to be one race: Node closed idle keep-alive connections after 5 seconds while its callers kept reusing them.
We moved our app to a new host and switched DNS on a Saturday. A day later almost half of requests still reached the old server. A 24-hour TTL, a nameserver change with a 48-hour delegation TTL and an old zone left online kept stale answers alive. This is the postmortem.
Our bot believed two orders were still open when they had filled hours earlier. The exchange closes websocket connections after 24 hours; our reconnect logic resubscribed but never checked what happened while disconnected. Reconnect with backoff plus REST reconciliation fixed the state drift.
After we tightened DMARC to quarantine, Gmail started filing our password resets and receipts as spam. Both SPF and DKIM passed, but for amazonses.com, not our domain. Here is how we read the headers, aligned both, and kept it that way.
Our live support chat dropped every connection that stayed quiet for a minute, with close code 1006 and no server error. Nginx's 60-second proxy_read_timeout and the load balancer idle timeout were closing idle sockets. A heartbeat plus proxy tuning fixed it.