Tag See all →

#aws-ecs

2 stories

Banner: latency climbed and the ECS service never scaled out, average CPU hid the saturated tasks

Cloud & Infrastructure Bug Report

Latency climbed and our ECS service never scaled out: average CPU hid the saturated tasks

Tomás Herrera · Jun 14, 2025 · 7 min read

During a flash sale our checkout API on ECS Fargate slowed to multi-second responses, yet the service stayed at six tasks. Target tracking on average CPU at 70% never fired: tasks were blocked on I/O, not CPU, and the max capacity was already reached on one service. How we chose better scaling signals.

Cloud & Infrastructure Bug Report

ECS scale-in killed SQS workers mid-job and jobs ran twice: task scale-in protection

Tomás Herrera · Jun 7, 2025 · 7 min read

Our ECS Fargate workers scaled on SQS queue depth. After each burst, scale-in stopped tasks in the middle of 10-minute PDF jobs; messages reappeared and the jobs ran again, sending customers duplicate documents. How we fixed it with task scale-in protection, SIGTERM handling and a visibility timeout that fits the work.