Latency climbed and our ECS service never scaled out: average CPU hid the saturated tasks

The marketing team launched a flash sale on a Friday at noon. Within minutes, the checkout API's p95 latency went from 180 ms to over 4 seconds, and the error rate rose as the load balancer timed out requests. The ECS service had target tracking autoscaling configured, and everyone assumed it would add tasks. It did not. The service stayed at six tasks for the whole first half hour.

What the scaling policy saw

aws application-autoscaling describe-scaling-policies --service-namespace ecs \
  --resource-id service/prod/checkout-api

"TargetTrackingScalingPolicyConfiguration": {
  "TargetValue": 70.0,
  "PredefinedMetricSpecification": { "PredefinedMetricType": "ECSServiceAverageCPUUtilization" },
  "ScaleOutCooldown": 300,
  "ScaleInCooldown": 300
}

And the metric during the incident: average CPU utilisation for the service hovered between 38% and 52%. Never above 70%. From the scaling policy's point of view, everything was fine.

Why CPU stayed low

The checkout API spends most of each request waiting: on the payment provider, on the inventory service, on the database. Under load, each task's Node.js process had hundreds of requests in flight, waiting on a connection pool of 20 database connections and an HTTP agent limited to 50 sockets to the payment provider. Requests queued inside the application. CPU was not the bottleneck; concurrency limits and downstream latency were.

The load balancer's metrics told the real story:

TargetResponseTime (p95)        4.2 s
RequestCountPerTarget           ~2,900 per minute per task (normal peak ~900)
HTTPCode_ELB_5XX_Count          rising

A second, quieter problem

When we raised capacity by hand, we hit another wall: the scalable target's MaxCapacity was 8. Someone had set it during initial setup to avoid runaway costs, and nobody had revisited it after traffic grew.

What we ruled out

• Scaling activity failing. There were no failed scaling activities; there were no scaling activities at all.
• Fargate capacity. Manual scaling to 8 tasks succeeded within two minutes.
• The alarm configuration. Target tracking creates its own CloudWatch alarms; they were in OK state because the metric was below target.

Root cause

The service scaled only on average CPU, which does not reflect load for an I/O-bound API. Requests queued inside tasks while CPU stayed moderate, so target tracking never triggered. And even correct scaling would have stopped at a maximum of eight tasks, too low for the sale.

Choosing a scaling metric

A good scaling metric rises proportionally with load and with the resource that runs out first.

For our checkout API, requests per target was the most direct signal: we knew from load tests that one task handled about 1,000 requests per minute within the latency target.

Cooldowns and scale-out speed

Target tracking scales out when the alarm fires, after the metric has been above target for the alarm's evaluation periods (several minutes for predefined metrics). The scale-out cooldown then blocks further scale-out activities for its duration, unless the alarm still demands more capacity, in which case target tracking can scale out again. Long cooldowns slow the reaction to a sharp spike.

Environment

Scale on requests per target

We replaced the CPU policy as primary with a request-based target tracking policy, keeping CPU as a second policy for protection:

resource "aws_appautoscaling_target" "checkout" {
  service_namespace  = "ecs"
  resource_id        = "service/prod/checkout-api"
  scalable_dimension = "ecs:service:DesiredCount"
  min_capacity       = 6
  max_capacity       = 40
}

resource "aws_appautoscaling_policy" "requests" {
  name               = "checkout-requests-per-target"
  policy_type        = "TargetTrackingScaling"
  service_namespace  = aws_appautoscaling_target.checkout.service_namespace
  resource_id        = aws_appautoscaling_target.checkout.resource_id
  scalable_dimension = aws_appautoscaling_target.checkout.scalable_dimension

  target_tracking_scaling_policy_configuration {
    target_value       = 700
    scale_out_cooldown = 60
    scale_in_cooldown  = 300
    predefined_metric_specification {
      predefined_metric_type = "ALBRequestCountPerTarget"
      resource_label         = "${aws_lb.main.arn_suffix}/${aws_lb_target_group.checkout.arn_suffix}"
    }
  }
}

resource "aws_appautoscaling_policy" "cpu" {
  name               = "checkout-cpu"
  policy_type        = "TargetTrackingScaling"
  service_namespace  = aws_appautoscaling_target.checkout.service_namespace
  resource_id        = aws_appautoscaling_target.checkout.resource_id
  scalable_dimension = aws_appautoscaling_target.checkout.scalable_dimension

  target_tracking_scaling_policy_configuration {
    target_value = 60
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }
  }
}

With several target tracking policies, Application Auto Scaling scales out if any policy asks for more capacity and scales in only when all policies agree. The target of 700 requests per minute per task leaves headroom below the measured capacity of about 1,000.

Raise the maximum, deliberately

Max capacity went from 8 to 40, based on load tests and on what the database could handle. Cost runaway is handled by a budget alarm and an alert when the service runs at more than 30 tasks, rather than by a cap low enough to cause an outage.

Scheduled scaling for known events

For planned sales, we now raise the minimum capacity ahead of time with a scheduled action, so capacity is in place before the traffic arrives:

aws application-autoscaling put-scheduled-action --service-namespace ecs \
  --resource-id service/prod/checkout-api --scalable-dimension ecs:service:DesiredCount \
  --scheduled-action-name flash-sale-2025-06-20 \
  --schedule "at(2025-06-20T11:30:00)" \
  --scalable-target-action MinCapacity=20,MaxCapacity=40

Fix the bottleneck inside the task

Scaling out helps only up to what downstream systems can take. We also raised the HTTP agent's socket limit to the payment provider within their documented limits and added a per-task concurrency limit that sheds load with a fast 503 instead of queueing requests for seconds.

Alternatives and trade-offs

• Step scaling on p95 latency. Latency reacts to every downstream problem, including ones more tasks cannot fix, which can cause pointless scale-out. Useful as an alert, risky as the only scaling signal.
• Custom in-flight request metric. More precise; requires publishing metrics from the application.

How to test the fix

A k6 load test ramps from 2,000 to 25,000 requests per minute over ten minutes against staging:

k6 run --vus 400 --duration 10m checkout-ramp.js
aws application-autoscaling describe-scaling-activities --service-namespace ecs \
  --resource-id service/staging/checkout-api --max-items 10

Expected: scale-out activities within two to three minutes of the ramp, p95 latency below 500 ms after new tasks are healthy.

Confirming in production

At the next sale, the scheduled action put 20 tasks in place before noon, the request policy scaled the service to 31 tasks at the peak, and p95 latency stayed under 300 ms. After the sale, the service scaled back to 6 tasks over about an hour.

Regression guard

• An alarm when desired count equals max capacity for more than ten minutes.
• An alarm on p95 TargetResponseTime independent of scaling metrics.
• Capacity planning review before every planned high-traffic event.

Security implications

None directly. Load shedding with fast 503s also limits the impact of abusive traffic, and the higher maximum capacity is paired with rate limiting at the edge, so an attack cannot simply buy itself 40 tasks of capacity on our bill.

Prevention checklist

• Scale on a metric that tracks your real bottleneck, not CPU by default.
• Keep CPU as a secondary policy, not the only one.
• Set max capacity from load tests, and alert when it is reached.
• Use scheduled scaling for known events.
• Shed load quickly inside tasks rather than queueing requests.

References

• Amazon ECS service auto scaling: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html
• Target tracking scaling policies for Application Auto Scaling: https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-target-tracking.html
• Scheduled scaling for Application Auto Scaling: https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-scheduled-scaling.html
• Application Load Balancer CloudWatch metrics: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-cloudwatch-metrics.html