ECS Fargate service keeps replacing tasks that fail ELB health checks

After a routine image update to our quotes-api service, the ECS console showed the deployment stuck at 0 of 3 healthy tasks. Tasks reached RUNNING, sat there for about half a minute, then stopped. The service events looked like a loop:

(service quotes-api) has started 1 tasks: (task 6f0d2a91c4e14b7a9f3c1e2d5a8b7c60).
(service quotes-api) registered 1 targets in (target-group arn:aws:elasticloadbalancing:eu-west-1:111122223333:targetgroup/quotes-api/8c1d5e2f9a0b3c47)
(service quotes-api) (task 6f0d2a91c4e14b7a9f3c1e2d5a8b7c60) failed ELB health checks in (target-group arn:aws:elasticloadbalancing:eu-west-1:111122223333:targetgroup/quotes-api/8c1d5e2f9a0b3c47)
(service quotes-api) has stopped 1 running tasks: (task 6f0d2a91c4e14b7a9f3c1e2d5a8b7c60).

The deployment circuit breaker eventually rolled back to the previous task definition, which was healthy. The new image worked fine when we ran it locally.

Reproducing it

Every deploy of the new image did the same thing; it was not intermittent. We pulled the stopped reason and the target health directly:

aws ecs describe-tasks --cluster prod --tasks 6f0d2a91c4e14b7a9f3c1e2d5a8b7c60 \
  --query 'tasks[0].{stop:stoppedReason,containers:containers[].{name:name,exit:exitCode}}'

{
  "stop": "Task failed ELB health checks in (target-group arn:aws:elasticloadbalancing:eu-west-1:111122223333:targetgroup/quotes-api/8c1d5e2f9a0b3c47)",
  "containers": [{ "name": "app", "exit": 143 }]
}

Exit code 143 is SIGTERM: ECS stopped a container that had not crashed on its own.

aws elbv2 describe-target-health \
  --target-group-arn arn:aws:elasticloadbalancing:eu-west-1:111122223333:targetgroup/quotes-api/8c1d5e2f9a0b3c47 \
  --query 'TargetHealthDescriptions[].[Target.Id,Target.Port,TargetHealth.State,TargetHealth.Reason]' \
  --output table

|  10.20.14.37 |  80 |  unhealthy |  Target.FailedHealthChecks |

Target.FailedHealthChecks is the reason the ALB reports when it cannot get a response at all, as opposed to Target.ResponseCodeMismatch (wrong status) or Target.Timeout.

What we ruled out

• App crash at startup. CloudWatch Logs showed a clean Spring Boot start and the line Tomcat started on port 8080 (http). No stack traces.
• Health endpoint broken. curl localhost:8080/actuator/health inside the container via ECS Exec returned {"status":"UP"}.
• Networking in general. The old task definition, in the same subnets and security groups, was healthy.

Root cause

Two problems, the second hidden behind the first.

The new image was built from our hardened base, which runs as a non-root user. A non-root process cannot bind port 80, so the team set SERVER_PORT=8080 in the image. The task definition, the service's loadBalancers block and the task security group still said port 80. The ALB was probing 10.20.14.37:80, where nothing listened. Connection refused, every time.

Once we fixed the port, tasks still flapped on roughly one deploy in three. The JVM took 55 to 75 seconds to become ready on 0.5 vCPU, and the service had no health check grace period. The target group marked targets unhealthy after two failed probes 15 seconds apart, so ECS was killing tasks that were simply still starting.

What is happening under the hood

With awsvpc networking, which Fargate always uses, every task gets its own ENI and the ALB registers the task's private IP plus the containerPort named in the service's loadBalancers configuration. The port inside the container has to match that number exactly; there is no host port mapping to translate.

ECS listens to the target group's verdict. If a target becomes unhealthy, the service scheduler stops that task and starts a replacement. healthCheckGracePeriodSeconds tells the scheduler to ignore unhealthy ELB results for a while after a task starts. It defaults to 0 when you create the service through the API or Terraform, which means the scheduler acts on the first unhealthy verdict.

The time to an unhealthy verdict is roughly interval x unhealthy_threshold. With our settings, 15 s x 2, that was about 30 seconds, well under our startup time.

Environment

Minimal reproduction

resource "aws_ecs_service" "api" {
  name            = "quotes-api"
  cluster         = aws_ecs_cluster.prod.id
  task_definition = aws_ecs_task_definition.api.arn
  desired_count   = 3
  launch_type     = "FARGATE"

  network_configuration {
    subnets         = var.private_subnet_ids
    security_groups = [aws_security_group.api_task.id]
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.api.arn
    container_name   = "app"
    container_port   = 80 # app now listens on 8080
  }
}

Combine that with an image that listens on 8080 and every task fails. Fix only the port, keep the 30-second unhealthy window and a 60-second startup, and you get the intermittent version.

Who is affected

Any ECS service behind an ALB or NLB, on Fargate or EC2 with awsvpc. Slow-starting runtimes (JVM, large Python apps that load models, .NET with heavy DI) are most exposed to the grace period problem. Port drift is common when a base image switches to a non-root user, since many official images moved to unprivileged ports such as 8080.

The fix

Port first. The container port appears in three places and all three must agree with what the process binds:

 # task definition container
 portMappings = [{
-  containerPort = 80
+  containerPort = 8080
   protocol      = "tcp"
 }]

 # service
 load_balancer {
   target_group_arn = aws_lb_target_group.api.arn
   container_name   = "app"
-  container_port   = 80
+  container_port   = 8080
 }

 # task security group: allow the ALB on the new port
 resource "aws_vpc_security_group_ingress_rule" "alb_to_task" {
   security_group_id            = aws_security_group.api_task.id
   referenced_security_group_id = aws_security_group.alb.id
   ip_protocol                  = "tcp"
-  from_port                    = 80
-  to_port                      = 80
+  from_port                    = 8080
+  to_port                      = 8080
 }

The target group itself keeps port = 80 as a default, but with health_check.port = "traffic-port" it probes whatever port ECS registers, so it did not need to change.

Then the grace period. We measured startup across 20 task launches (time from task start to the first UP response) and saw a p100 of 78 seconds. We set the grace period to 120 seconds to leave headroom:

 resource "aws_ecs_service" "api" {
   ...
+  health_check_grace_period_seconds = 120
 }

We also made startup itself faster by moving to 1 vCPU, which brought it to about 35 seconds. The grace period stays as a safety margin, not a substitute.

Alternatives and trade-offs

• Raise the unhealthy threshold instead. Works, but it also delays detection of a task that dies later in life. The grace period only affects the start window.
• Container health check in the task definition. Useful for non-load-balanced services and for dependency ordering, but beware that a curl based check fails silently on distroless images that have no curl. ECS then reports Task failed container health checks, a different message from the ELB one.
• Spring Boot readiness probe path. Pointing the target group at /actuator/health/readiness keeps tasks out of rotation until the app reports ready, which pairs well with a grace period.

How to test the fix

# Is the app listening where the ALB will probe? Run inside the task via ECS Exec.
aws ecs execute-command --cluster prod --task <task-id> --container app \
  --interactive --command "sh -c 'ss -ltn | grep 8080'"

# Deploy and watch the targets settle
aws ecs update-service --cluster prod --service quotes-api --force-new-deployment
aws ecs wait services-stable --cluster prod --services quotes-api && echo stable

aws elbv2 describe-target-health --target-group-arn "$TG_ARN" \
  --query 'TargetHealthDescriptions[].TargetHealth.State'
# expected: ["healthy", "healthy", "healthy"]

services-stable waits until the running count equals the desired count with a single deployment, and gives up after 40 checks 15 seconds apart.

Confirming in production

We ran ten consecutive forced deployments and checked that none produced a failed ELB health checks event:

aws ecs describe-services --cluster prod --services quotes-api \
  --query 'services[0].events[?contains(message, `failed ELB health checks`)] | length(@)'

We also added a CloudWatch alarm on the target group's UnHealthyHostCount (Maximum over 5 minutes greater than 0, three datapoints) and on the ECS service's running task count dropping below the desired count.

Regression guard

A precondition in the service module now fails the plan if the load balancer port and the port mapping drift apart, so the two cannot diverge in a review. We chose precondition over a check block because a failed check only produces a warning:

locals {
  mapped_port = one(jsondecode(aws_ecs_task_definition.api.container_definitions)[0].portMappings).containerPort
}

resource "aws_ecs_service" "api" {
  # ...
  lifecycle {
    precondition {
      condition     = var.container_port == local.mapped_port
      error_message = "load_balancer container_port must match the task definition portMappings."
    }
  }
}

The image build also records the listening port as a label, and the deploy pipeline compares it with the task definition before calling update-service.

Security implications

The port change was a security improvement: the process no longer needs root to bind a privileged port. When fixing it, we opened the task security group only to the ALB's security group on 8080, not to the VPC CIDR. Watch out for the tempting shortcut of allowing 0.0.0.0/0 on the task port to "rule out networking"; tasks in public subnets with public IPs would then be reachable directly, bypassing the ALB, its WAF rules and TLS termination. The health endpoint exposes only the aggregate status; we kept management.endpoint.health.show-details=never so it does not leak dependency names to anyone who can reach it.

Prevention checklist

• Keep container port, service load_balancer port and the task security group rule in one variable.
• Measure real startup time and set health_check_grace_period_seconds above it.
• Use traffic-port for target group health checks so they follow the registered port.
• Read TargetHealth.Reason before guessing: connection failure, wrong code and timeout have different causes.
• Enable the deployment circuit breaker with rollback so a bad deploy does not run unattended.

References

• Amazon ECS, health checks for services behind a load balancer: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/load-balancer-healthcheck.html
• Amazon ECS API, healthCheckGracePeriodSeconds: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_CreateService.html
• Elastic Load Balancing, target health reason codes: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html
• Amazon ECS, deployment circuit breaker: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-circuit-breaker.html
• Terraform AWS provider, aws_ecs_service: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ecs_service