ECS scale-in killed SQS workers mid-job and jobs ran twice: task scale-in protection

Customers of our document service started receiving the same generated contract twice, sometimes three times. The service is simple: an API puts a job on an SQS queue, and worker tasks on ECS Fargate take jobs, render a PDF (5 to 12 minutes for large contracts), upload it and email the customer. Workers scale with Application Auto Scaling on queue depth.

The duplicates clustered right after busy periods. Logs for one duplicated job:

10:02:11 worker task/6f1c... job=c_7781 started
10:06:40 worker task/6f1c... received SIGTERM, shutting down
10:06:42 worker task/6f1c... exited (code 143)
10:14:12 worker task/9a3e... job=c_7781 started            <- same job, new task
10:22:57 worker task/9a3e... job=c_7781 finished, email sent

The first task was stopped mid-job. And one more case showed both tasks sending the email.

Why were tasks stopped?

ECS service events showed scale-in activity at the same times:

service doc-workers has stopped 4 running tasks: task 6f1c..., task 81d2..., ...
(service doc-workers) has reached a steady state.

The scaling policy scaled in when queue depth dropped. ECS chose which tasks to stop without knowing which were busy, and stopped tasks holding in-flight messages.

Why did the job run again, and sometimes twice?

Two mechanisms:

1. The stopped worker had received the message but not deleted it. After the queue's visibility timeout expired, the message became visible again and another worker picked it up. Correct at-least-once behaviour.
2. In the "both emailed" case, the visibility timeout was 5 minutes, shorter than a 9-minute job. The message became visible again while the first worker was still working, and a second worker started the same job in parallel. That one had nothing to do with scale-in; it was a latent bug made more frequent by the retries.

What we ruled out

• Duplicate enqueueing by the API. Each job id appeared once in the API logs and once in SendMessage CloudTrail events.
• Worker crashes. The exit code was 143, meaning terminated by SIGTERM, not an application error.
• SQS FIFO deduplication. The queue was standard; FIFO deduplication would not have helped with re-delivery after timeout anyway.

Root cause

Scale-in stopped worker tasks without regard to work in progress, and the workers did not finish or release their current job on SIGTERM within the default 30-second stop timeout. Separately, the queue's visibility timeout was shorter than the longest job, which allowed concurrent duplicate processing.

How ECS stops tasks on scale-in

When a service's desired count decreases, ECS selects tasks to stop, considering placement strategy across Availability Zones, and sends each container SIGTERM. After stopTimeout (30 seconds by default on Fargate, configurable up to 120 seconds), it sends SIGKILL. A worker in the middle of a 10-minute job cannot finish in 30 seconds.

Task scale-in protection

ECS task scale-in protection lets a task mark itself as protected. Protected tasks are not chosen when the service scales in or during deployments, until protection expires or the task removes it. The task calls an agent endpoint available inside the container, $ECS_AGENT_URI/task-protection/v1/state.

SQS visibility timeout

When a consumer receives a message, SQS hides it for the visibility timeout. If the consumer does not delete it in time, the message becomes visible again. For long jobs, either set the timeout above the longest job, or extend it periodically with ChangeMessageVisibility while working.

Environment

Step 1: protect the task while it works

import os, requests

AGENT = os.environ["ECS_AGENT_URI"]

def set_protection(enabled: bool, minutes: int = 20):
    body = {"ProtectionEnabled": enabled}
    if enabled:
        body["ExpiresInMinutes"] = minutes
    r = requests.put(f"{AGENT}/task-protection/v1/state", json=body, timeout=2)
    r.raise_for_status()

def work_loop():
    while not shutting_down:
        msgs = sqs.receive_message(QueueUrl=QUEUE, MaxNumberOfMessages=1, WaitTimeSeconds=20,
                                   VisibilityTimeout=900).get("Messages", [])
        if not msgs:
            continue
        msg = msgs[0]
        set_protection(True, minutes=20)
        try:
            process(msg)
            sqs.delete_message(QueueUrl=QUEUE, ReceiptHandle=msg["ReceiptHandle"])
        finally:
            set_protection(False)

Protection is set only while a job is in progress, with an expiry that exceeds the longest job, so a hung task cannot block scale-in forever. Idle tasks are unprotected and can be stopped at any time.

The task role needs no extra IAM permissions for this endpoint; it is served by the ECS agent.

Step 2: handle SIGTERM gracefully

SIGTERM can still arrive (deployments with a protected task wait, but other events do not). The worker now stops taking new messages on SIGTERM and either finishes the current job or returns the message:

import signal

shutting_down = False

def on_sigterm(signum, frame):
    global shutting_down
    shutting_down = True

signal.signal(signal.SIGTERM, on_sigterm)

If a job cannot finish before stopTimeout, the worker sets the message's visibility timeout to 0, so another worker can pick it up immediately instead of waiting 15 minutes. We raised stopTimeout to 120 seconds in the task definition to give short jobs a chance to complete.

Step 3: visibility timeout longer than the work

The queue's visibility timeout is now 15 minutes, above the longest observed job. For safety against unusually slow jobs, a background thread extends the visibility every five minutes while processing:

sqs.change_message_visibility(QueueUrl=QUEUE, ReceiptHandle=handle, VisibilityTimeout=900)

Step 4: make the effects idempotent

Even with all of the above, SQS standard queues deliver at least once. The final email step now records (job_id, 'emailed') in DynamoDB with a conditional write before sending. A second attempt finds the record and skips the email; the PDF upload to a deterministic S3 key is naturally idempotent.

Alternatives and trade-offs

• AWS Batch or Step Functions with ECS tasks for long jobs: one task per job, nothing to scale in mid-job.
• Scale in more conservatively with longer cooldowns. Reduces frequency, does not fix the problem.

How to test the fix

On staging, enqueue 50 long jobs, let workers scale out, then drain the queue and watch scale-in:

aws ecs describe-tasks --cluster docs --tasks $(aws ecs list-tasks --cluster docs --service-name doc-workers \
  --query 'taskArns' --output text) --query 'tasks[].[taskArn,lastStatus]'
curl -s "$ECS_AGENT_URI/task-protection/v1/state"     # from inside a busy task: ProtectionEnabled true

Expected: ECS stops only idle tasks, and every job id appears exactly once in "finished" logs.

Confirming in production

Over the following month, with similar traffic, zero duplicate documents were reported. Logs showed SIGTERMs only for idle tasks during scale-in. Scale-in took slightly longer on average, because busy tasks were kept until their jobs completed.

Regression guard

• A metric counts jobs started more than once per job id; an alarm fires on any value above zero.
• Alarm on the queue's ApproximateAgeOfOldestMessage, so protection that blocks scale-in too long, or stuck workers, is visible.

Security implications

None directly. The duplicate emails did send contract documents twice to the correct recipients, so no data went to the wrong person. Idempotency keys also protect against accidental double sends after future failures.

Prevention checklist

• Use ECS task scale-in protection while a worker processes a long job.
• Handle SIGTERM: stop receiving, finish or release the current message.
• Set visibility timeout above the longest job, or extend it while working.
• Make side effects idempotent; at-least-once delivery will happen.
• Monitor duplicate processing explicitly.

References

• Protect Amazon ECS tasks from being terminated by scale-in events: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-scale-in-protection.html
• Task scale-in protection endpoint: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-scale-in-protection-endpoint.html
• Amazon SQS visibility timeout: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
• ECS task definition stopTimeout: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html