SQS messages processed twice: visibility timeout shorter than the Lambda timeout

Customers on our invoicing product started reporting two identical "Your invoice is ready" emails, each with the same PDF attached. Support counted about 1 in 60 invoices affected, almost all of them large ones with many line items. The duplicates arrived a minute or two apart.

The render-invoice Lambda logs showed the same SQS message handled by two invocations:

2026-09-01T10:14:03.221Z  a41c...  INFO  start messageId=6f1e2c0a-91b7-4c1e-9b8d-3f0a7d2e5c11 invoice=inv_48812 receiveCount=1
2026-09-01T10:15:03.884Z  c9d7...  INFO  start messageId=6f1e2c0a-91b7-4c1e-9b8d-3f0a7d2e5c11 invoice=inv_48812 receiveCount=2
2026-09-01T10:15:31.402Z  a41c...  INFO  done  invoice=inv_48812 durationMs=88181
2026-09-01T10:16:20.517Z  c9d7...  INFO  done  invoice=inv_48812 durationMs=76633

The second start came almost exactly 60 seconds after the first.

Reproducing it

The pipeline: the billing API sends a message to the standard queue invoice-render when an invoice is finalised. A Lambda event source mapping polls the queue with batch size 5. The function renders a PDF, uploads it to S3 and calls the email service.

Two weeks earlier we had raised the function timeout from 60 to 180 seconds because big invoices were timing out. To reproduce, we sent a message for a synthetic 4,000-line invoice that takes about 90 seconds to render. It was processed twice every time.

What we ruled out

• SQS at-least-once delivery. Standard queues can occasionally deliver a message twice, but that is rare and random. Ours was systematic, correlated with render time, and showed receiveCount=2.
• Duplicate sends from the API. Each case had one SendMessage in CloudTrail data events and one message id.
• Lambda retries after errors. Both invocations finished successfully. Nothing failed.

Root cause

When Lambda receives a message from SQS, the message stays in the queue but becomes invisible for the queue's visibility timeout. Lambda deletes it only after the function returns successfully. Our queue's VisibilityTimeout was 60 seconds. Renders longer than 60 seconds let the message become visible again while the first invocation was still working, and the poller handed it to a second invocation. When AWS created the event source mapping, the function timeout was 60 seconds, so the settings matched; raising the function timeout later did not touch the queue.

What is happening under the hood

SQS has no idea a consumer is still working. The visibility timeout is a lease: once it expires, the message is fair game for any receiver. Lambda's SQS poller runs several concurrent pollers and scales invocations with queue depth, so a message that reappears is picked up within seconds.

AWS checks the relationship only at one point. Creating an event source mapping against a queue whose visibility timeout is shorter than the function timeout fails:

An error occurred (InvalidParameterValueException) when calling the CreateEventSourceMapping operation: Queue visibility timeout: 30 seconds is less than Function timeout: 60 seconds

In our account, raising the function timeout afterwards with update-function-configuration succeeded without any warning, and that is how the two values drifted apart.

The AWS guidance for Lambda with SQS is to set the queue visibility timeout to at least six times the function timeout, plus the value of MaximumBatchingWindowInSeconds. The margin covers throttling retries: if Lambda is throttled while invoking with a batch, it keeps retrying within the visibility window.

Batch size makes it worse. All messages in a batch share one invocation, so a batch with one slow invoice keeps the other four in flight too. If the invocation runs past the visibility timeout, all five can reappear.

Environment

Minimal reproduction

# template.yaml (excerpt, broken)
Resources:
  RenderFn:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: nodejs22.x
      Handler: index.handler
      Timeout: 180
      Events:
        Queue:
          Type: SQS
          Properties:
            Queue: !ImportValue invoice-render-queue-arn
            BatchSize: 5
            MaximumBatchingWindowInSeconds: 2

// index.mjs
export const handler = async (event) => {
  for (const record of event.Records) {
    const { invoiceId } = JSON.parse(record.body);
    console.log(`start messageId=${record.messageId} invoice=${invoiceId} receiveCount=${record.attributes.ApproximateReceiveCount}`);
    await new Promise((r) => setTimeout(r, 90_000)); // stands in for a slow render
    console.log(`done invoice=${invoiceId}`);
  }
};

Send one message and watch the logs; receiveCount=2 appears after 60 seconds.

Who is affected

Any SQS consumer whose processing time can exceed the queue visibility timeout: Lambda, ECS workers, or anything calling ReceiveMessage. It is especially common when the queue and the function live in different stacks or are owned by different teams, so a timeout change in one does not prompt a change in the other.

The fix

Three changes: correct the timeouts, report partial batch failures, and make the handler idempotent.

1. Visibility timeout from the function timeout. We moved the queue into the same stack and derived the value instead of hard-coding it:

 Parameters:
   RenderTimeout:
     Type: Number
     Default: 180
+  RenderVisibilityTimeout:
+    Type: Number          # computed by deploy script: 6 x RenderTimeout + window + margin
+    Default: 1110
 Resources:
   RenderQueue:
     Type: AWS::SQS::Queue
     Properties:
-      VisibilityTimeout: 60
+      VisibilityTimeout: !Ref RenderVisibilityTimeout
       RedrivePolicy:
         deadLetterTargetArn: !GetAtt RenderDLQ.Arn
         maxReceiveCount: 5
   RenderFn:
     Type: AWS::Serverless::Function
     Properties:
-      Timeout: 180
+      Timeout: !Ref RenderTimeout

CloudFormation does not do arithmetic on parameters, so the deploy script computes the visibility value (6 x 180 s, plus the 2 s batching window, plus a margin) and passes both in; a unit test asserts the relationship (shown below).

2. Smaller batches and partial failures. We dropped BatchSize to 1 for this queue, since renders are heavy and per-message isolation matters more than throughput. For other queues where we keep larger batches, we enable ReportBatchItemFailures so only failed messages return to the queue:

      Events:
        Queue:
          Type: SQS
          Properties:
            Queue: !GetAtt RenderQueue.Arn
            BatchSize: 1
            FunctionResponseTypes:
              - ReportBatchItemFailures

export const handler = async (event) => {
  const batchItemFailures = [];
  for (const record of event.Records) {
    try {
      await renderAndSend(JSON.parse(record.body));
    } catch (err) {
      console.error(`failed messageId=${record.messageId}`, err);
      batchItemFailures.push({ itemIdentifier: record.messageId });
    }
  }
  return { batchItemFailures };
};

3. Idempotent processing. Even with correct timeouts, standard queues are at-least-once. We wrapped the render with the Powertools idempotency utility, keyed on the invoice id and backed by DynamoDB:

import { makeIdempotent, IdempotencyConfig } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';

const persistenceStore = new DynamoDBPersistenceLayer({ tableName: 'render-idempotency' });
const config = new IdempotencyConfig({ eventKeyJmesPath: 'invoiceId', expiresAfterSeconds: 86400 });

const renderAndSend = makeIdempotent(
  async (payload) => {
    const pdfKey = await renderPdf(payload.invoiceId);
    await emailService.sendInvoiceReady(payload.invoiceId, pdfKey);
    return { pdfKey };
  },
  { persistenceStore, config },
);

A second invocation for the same invoice now returns the stored result instead of rendering and emailing again. While the first is still running, the second sees an in-progress record and throws, which sends the message back to the queue for a later retry.

Alternatives and trade-offs

• Extend visibility while working with ChangeMessageVisibility as a heartbeat. Necessary for open-ended jobs in custom workers; with Lambda the function timeout already bounds the work, so a static value is simpler.
• SQS FIFO with deduplication prevents duplicate sends within five minutes but does not stop redelivery after a visibility timeout expires.
• Move rendering to Step Functions or ECS if renders keep growing toward the 15-minute Lambda limit.

How to test the fix

A guard test that fails the build when the values drift:

// test/timeouts.test.mjs
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { readFileSync } from 'node:fs';

test('queue visibility timeout covers 6x function timeout', () => {
  const cfg = JSON.parse(readFileSync('deploy/render.params.json', 'utf8'));
  assert.ok(cfg.VisibilityTimeout >= 6 * cfg.RenderTimeout + cfg.BatchingWindow,
    `visibility ${cfg.VisibilityTimeout}s too short for timeout ${cfg.RenderTimeout}s`);
});

node --test test/
aws sqs send-message --queue-url https://sqs.eu-west-1.amazonaws.com/123456789012/invoice-render \
  --message-body '{"invoiceId":"inv_TEST_4000_LINES"}'

The 90-second render now logs one start with receiveCount=1 and one done.

Confirming in production

Check the live values first:

aws sqs get-queue-attributes --queue-url "$QUEUE_URL" --attribute-names VisibilityTimeout
aws lambda get-function-configuration --function-name render-invoice --query Timeout

Then compare messages received with messages deleted in CloudWatch. NumberOfMessagesReceived was about 4 percent higher than NumberOfMessagesSent before the fix; afterwards the two track within normal at-least-once noise. We also count log lines with receiveCount greater than 1 using a metric filter; the daily count went from about 150 to under 5, and none of those produced a second email because of the idempotency layer. Duplicate-email support tickets stopped the same week.

Regression guard

• The timeout relationship test runs in CI for every stack that wires a function to a queue.
• An AWS Config custom rule flags any Lambda SQS mapping where queue visibility is below six times the function timeout.
• Alarm on ApproximateAgeOfOldestMessage and on any messages in the DLQ.

Security implications

Duplicate emails leaked nothing new, since both went to the right recipient, but repeated emails train customers to ignore them and help phishing look normal. A visibility timeout that is too short also amplifies load: a slow dependency causes redelivery, which adds more slow work. The idempotency table stores invoice ids and S3 keys only, with a 24-hour TTL; the function role has dynamodb:GetItem, PutItem, UpdateItem and DeleteItem on that one table.

Prevention checklist

• Set queue visibility timeout to at least six times the function timeout plus the batching window.
• Define the queue and its consumer in the same stack, or derive one value from the other.
• Use ReportBatchItemFailures for batches larger than one.
• Make SQS consumers idempotent regardless of timeouts.
• Alert on ApproximateReceiveCount above 1 for queues where duplicates are costly.

References

• AWS Lambda Developer Guide, Using Lambda with Amazon SQS: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
• Amazon SQS Developer Guide, Visibility timeout: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
• AWS Lambda Developer Guide, Handling errors for an SQS event source: https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-errorhandling.html
• Powertools for AWS Lambda (TypeScript), Idempotency: https://docs.powertools.aws.dev/lambda/typescript/latest/features/idempotency/