Cloud services, Kubernetes, servers, storage, cost surprises.
Video and dataset uploads of 5 to 15 GB kept failing through our API servers. We moved to browser-direct S3 multipart uploads with per-part presigned URLs, resumable state, per-part checksums and server-side completion. This is the full design and working code.
Commitment discounts are the biggest lever on a steady AWS bill and the easiest way to lock in waste. This guide explains Compute and EC2 Instance Savings Plans versus Reserved Instances, coverage versus utilisation, laddering purchases, and when not to commit.
We added a new GPU node pool and training pods stayed Pending with Insufficient nvidia.com/gpu, while the nodes sat idle. The NVIDIA device plugin was not running on the new nodes because its DaemonSet did not tolerate their taint, so the nodes never advertised GPUs.
EBS storage kept rising while our clusters stayed the same size. Preview environments created PersistentVolumes with a Retain policy, deleted namespaces left the volumes behind, and a snapshot job never expired anything. We found, verified and deleted 612 volumes and 9,400 snapshots.
A new module for DNS validation of ACM certificates failed at plan time with an Invalid for_each argument error on fresh environments. The for_each keys came from attributes only known after apply. Static keys with apply-time values made the plan work in one pass.
Receipt uploads to S3 failed with 403 SignatureDoesNotMatch for about 4 percent of users, mostly iPhone photos uploaded from Windows. The URL was signed with one Content-Type and the browser sent another. Reading the CanonicalRequest in the error pinpointed it.
Our first Terraform apply in ap-southeast-3 failed with an explicit deny even though the role had AdministratorAccess. The cause was the Control Tower Region deny control. Here is how we proved it, enabled the region the supported way, and redesigned region governance in our landing zone.
We moved twelve services from IRSA to EKS Pod Identity to simplify IAM. Three of them started failing with AccessDenied or fell back to the node role. Causes: a missing agent add-on on one node group, an old AWS SDK that did not know the new credential provider, and a trust policy without sts:TagSession.
A cancelled GitLab pipeline killed terraform apply mid-run and left the S3 backend locked. Every plan after that failed with Error acquiring the state lock. How we proved no run was active, unlocked safely, checked for drift, and stopped CI from cancelling applies.
Our order fulfilment state machine ran for months, then failed for large orders with States.DataLimitExceeded. A Map state collected per-item results into one array that crossed the 256 KiB payload limit. How we found the state that grew, trimmed outputs with ResultSelector and moved bulk data to S3.
A shared Kubernetes cluster shows up on the cloud bill as a few lines of EC2 and EBS. Turning that into cost per team needs decisions: allocate by requests or usage, what to do with idle capacity, and how to split shared services. This guide covers the model we use and why.
After a node group replacement, a Redis StatefulSet pod stayed Pending with 1 node(s) had volume node affinity conflict. Its EBS volume lived in eu-west-1a, and no node in that zone had room. How zonal volumes, WaitForFirstConsumer and topology interact, and how we made the StatefulSet survive node changes.
We ran everything in one big Kubernetes cluster until a CRD upgrade broke three teams at once. This guide compares one shared cluster, a cluster per environment and a cluster per team on isolation, blast radius, upgrade effort and real cost, with a worked cost model.
After upgrading to Karpenter v1 with WhenEmptyOrUnderutilized consolidation, our EKS cluster replaced nodes constantly and long-running pods were evicted every few minutes. How we read Karpenter's events, tuned consolidateAfter and disruption budgets, and protected jobs that must not be interrupted.
Every deploy of our Go pricing API produced 30 to 60 seconds of 502 and 503 errors. New pods got traffic before they were ready, and old pods exited while the ingress still routed to them. A readiness probe, a preStop sleep and real graceful shutdown made rollouts clean.
We deleted one queue name from a Terraform list and the plan wanted to destroy and recreate every queue after it. count addresses resources by position; we moved to for_each with moved blocks and changed nothing in AWS.
Our analytics VPC could reach the shared services VPC, and shared services could reach the payments VPC, but analytics timed out connecting to payments. VPC peering does not route traffic through a middle VPC. How we proved it, the options, and why we moved to Transit Gateway.
Our self-managed cluster pulled images from ECR fine after setup and failed every morning with no basic auth credentials. The imagePullSecret held an ECR token that expires after 12 hours. We switched to the kubelet credential provider so nodes fetch fresh tokens themselves.
An overview of Cloudflare's Free, Pro, Business and Enterprise plans, then a practical guide to everything the free tier offers a small team: DNS, CDN, Universal SSL, DDoS protection, WAF managed rules, rate limiting, Tunnel, Zero Trust, Pages, Workers, R2 and Turnstile, with a setup order.
Our single-host production server stopped accepting writes at 03:12 with no space left on device. Docker had quietly filled the disk with a 31 GB container log, old images and build cache. Here is how we found each piece and stopped it from coming back.
Our app said the media bucket held 4 TB, while S3 billed us for 31 TB. Versioning kept every overwritten and deleted object as a noncurrent version, and failed multipart uploads left invisible parts behind. Lifecycle rules brought billed storage down to 5.2 TB.
A hotfix image pushed from an M3 laptop crash-looped on our x86 servers with exec format error. The image was linux/arm64 only. We moved builds to CI with buildx multi-platform manifests and stopped laptop pushes to the registry.
Our storefront BFF pods restarted every few hours with OOMKilled and exit code 137, then fell into CrashLoopBackOff during the holiday peak. A max-old-space-size inherited from the VM days was double the container memory limit. Sizing the heap to the limit and bounding a cache fixed it.
We put SNS in front of an SQS queue to fan out order events, and the consumer Lambda started failing with KeyError: 'orderId'. Each SQS message body was now an SNS envelope with our event as an escaped JSON string in Message. How raw message delivery fixes it, and what changes for filtering and attributes.
Our CloudWatch line went from 2,100 to 6,900 USD in a month without new services. Debug logging enabled during an incident was never turned off, and every load balancer health check produced a log line. Finding the top log groups and fixing levels cut ingestion by 81 percent.