Our primary region could not write to its database for 38 minutes, yet Route 53 kept answering with the primary endpoint. The health check was green the whole time. Then automatic failback broke writes a second time. Here is why, and the health check design that fixed both.
Backup and restore, pilot light, warm standby and active-active are not maturity levels. They are price points for a given RPO and RTO. Here is how we picked a different strategy for each of five workloads, what each one cost, and how we prove the numbers.
We enabled KMS encryption for Session Manager to meet an audit requirement and every session to our EC2 fleet failed. The instance role could not use the key, the key policy did not allow it, and private instances could not reach KMS at all. The full fix and checks.
Our Terraform repo grew to two clouds, four environments and 140 state files, and every change felt risky. This is the layout we moved to: versioned modules, a unit catalog, Terragrunt stacks per region, and one state file per small blast radius.
EKS Auto Mode hands node provisioning, core add-ons, load balancing and block storage to AWS. We moved two clusters from managed node groups and self-managed Karpenter. What Auto Mode manages, what changes for operators, the limits that surprised us, and the migration steps that kept workloads running.
Our first Site-to-Site VPN between an office firewall and AWS came up green on both sides, yet pings failed and, once they worked, large file transfers hung. Three separate problems: route propagation, the firewall's traffic selectors and MTU. How we found each one.
Our order worker stopped completely for 47 minutes and the throughput alarm never fired. With no datapoints arriving, the alarm moved to INSUFFICIENT_DATA instead of ALARM, and only ALARM had an action. The fix was choosing missing data treatment per alarm.
Moving an on-premises RHEL 9 VM to EC2 with VM Import failed twice: first on the VMDK format, then on GRUB. Once the AMI finally booted, it had no network. Here is the pre-flight checklist we now run on every guest before uploading a disk.
One Terraform state for a whole environment meant 11-minute plans, fragile applies and an app deploy that could touch the VPC. How we split it into layers by rate of change and blast radius, passed values between them, moved resources with removed and import blocks, and kept the apply order sane.
Order events from our platform account were supposed to reach an SQS queue owned by the analytics account. The rule showed matches, the queue showed nothing, and no error appeared anywhere obvious. Causes: a queue policy without the rule ARN, a KMS key policy that denied EventBridge, and no DLQ to tell us.
Instances in a freshly added private subnet timed out on dnf and never showed up in Session Manager, while identical instances in older subnets were fine. The subnet had silently fallen back to the VPC main route table, which had no NAT route.
Our ECS service started, registered, failed health checks and got replaced, over and over. Two separate problems stacked: the container moved to port 8080 while the target group still probed 80, and a slow JVM start ran out the clock with no grace period.
A researcher reported that CSV exports with customer emails were downloadable from an S3 bucket. The cause was a bucket policy granting public read, added years earlier for a static site. How we contained it, proved who accessed what, and locked down every bucket.
A security group rule added in the console during an incident survived for eight months and was wiped out by an unrelated apply. Drift is normal; unmanaged drift is the problem. This guide covers scheduled drift detection, triage, import blocks and guardrails against click-ops.
We inherited an AWS account built by hand: 60 resources nobody dared to touch. Terraform import blocks with -generate-config-out let us adopt them in reviewed pull requests with a zero-change plan as proof. The workflow, the cleanup the generated code needs, and the traps.
A practitioner's reading of AWS's official summary of the October 19-20, 2025 us-east-1 disruption: a race in DynamoDB's DNS automation left an empty record, and recovery cascaded through EC2, NLB and Lambda. What happened per the report, and what we changed.
We moved our EKS batch node pool to Spot and cut its cost by 60 percent. Within a week, nightly jobs failed most nights: a single instance type, no interruption handling and jobs with no checkpoints. Diversified instances, graceful draining and checkpointing made Spot reliable.
Terraform's S3 backend can lock state with a lock file in the bucket itself, and DynamoDB-based locking is deprecated. How S3 native locking works, the IAM it needs, a safe migration with both mechanisms on during the transition, and the failure modes we tested.
Our AWS bill jumped by over 9,000 USD in one month and nothing looked broken. The cause was NAT Gateway data processing: batch jobs in private subnets were reading S3 and pulling ECR images through NAT. VPC endpoints cut that line item by 94 percent.
We spent a quarter tagging every resource with team and service, then grouped Cost Explorer by tag and saw 92 percent under No tag key. User-defined tags must be activated as cost allocation tags, they apply from activation onward, and some spend cannot be tagged at all.
Tenants upload terabytes of audio, video, PDFs and images, and expect an LLM to answer questions over all of it, with their data isolated and a bill that matches what they used. Here is the AWS design we shipped: ingestion pipeline, isolation layers and metering.
An email provider outage stopped customers from placing orders. Nothing about orders needed email. We rebuilt the service around three ideas: static stability, cell-based isolation and a blast-radius budget per dependency. Here is the worked design.
We moved all EBS volumes from gp2 to gp3 to save 20 percent, as every cost guide recommends. Two days later, nightly reports on our self-managed Postgres took twice as long. The 2 TB gp2 volume had 6,000 IOPS and 250 MiB/s; gp3 defaults are 3,000 and 125. Provisioning both fixed it.
Our WAF rate limit and managed rules worked perfectly, for anyone polite enough to go through CloudFront. A scripted client hitting the ALB hostname directly skipped all of it. Here is how we found it, why it happens, and the layered origin protection we now ship by default.
Commitment discounts are the biggest lever on a steady AWS bill and the easiest way to lock in waste. This guide explains Compute and EC2 Instance Savings Plans versus Reserved Instances, coverage versus utilisation, laddering purchases, and when not to commit.