Our order worker stopped completely for 47 minutes and the throughput alarm never fired. With no datapoints arriving, the alarm moved to INSUFFICIENT_DATA instead of ALARM, and only ALARM had an action. The fix was choosing missing data treatment per alarm.
A misconfigured batch job launched 60 GPU instances on a Friday evening. Our monthly budget alert fired on Sunday afternoon, after 14,000 USD was spent. Monthly thresholds and billing data lag make budgets slow. We added daily budgets, anomaly detection and real-time usage guardrails.
During a flash sale our checkout API on ECS Fargate slowed to multi-second responses, yet the service stayed at six tasks. Target tracking on average CPU at 70% never fired: tasks were blocked on I/O, not CPU, and the max capacity was already reached on one service. How we chose better scaling signals.
Our CloudWatch line went from 2,100 to 6,900 USD in a month without new services. Debug logging enabled during an incident was never turned off, and every load balancer health check produced a log line. Finding the top log groups and fixing levels cut ingestion by 81 percent.