SNS to SQS fan-out delivered our JSON as an escaped string inside an envelope: raw message delivery

For a year, the order service sent events straight to an SQS queue consumed by a Lambda function that updates a reporting table. Then two more teams wanted the same events, so we introduced an SNS topic: the order service publishes to the topic, and each team subscribes its own queue. The publisher code changed from sqs.send_message to sns.publish with the same JSON. The existing consumer immediately started failing:

[ERROR] KeyError: 'orderId'
Traceback (most recent call last):
  File "/var/task/handler.py", line 14, in handler
    order_id = event_body["orderId"]

What the consumer now received

We logged one raw SQS record:

{
  "messageId": "1f0c3e4a-...",
  "body": "{\n  \"Type\" : \"Notification\",\n  \"MessageId\" : \"7b2d...\",\n  \"TopicArn\" : \"arn:aws:sns:eu-west-1:111122223333:order-events\",\n  \"Message\" : \"{\\\"orderId\\\":\\\"o_88123\\\",\\\"total\\\":129.9,\\\"currency\\\":\\\"EUR\\\"}\",\n  \"Timestamp\" : \"2024-12-05T10:14:02.117Z\",\n  \"SignatureVersion\" : \"1\",\n  \"Signature\" : \"...\",\n  \"SigningCertURL\" : \"https://sns.eu-west-1.amazonaws.com/...\",\n  \"UnsubscribeURL\" : \"https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&...\",\n  \"MessageAttributes\" : { \"eventType\" : { \"Type\" : \"String\", \"Value\" : \"OrderPlaced\" } }\n}",
  "attributes": { "ApproximateReceiveCount": "1", "SentTimestamp": "1733393642117" },
  "messageAttributes": {}
}

Three layers: the SQS record, whose body is a JSON string of the SNS notification envelope, whose Message field is a JSON string of our event. Our handler parsed body once and looked for orderId in the envelope.

The other thing to notice: messageAttributes on the SQS record was empty. The attributes we set when publishing to SNS were inside the envelope, not on the SQS message.

What we ruled out

• Serialisation changes in the publisher. The publisher passed the same json.dumps(event) string; the extra encoding happened in SNS delivery.
• A misconfigured subscription protocol. The subscription protocol was sqs, which is correct.
• Batch size or partial batch handling. Every record failed the same way.

Root cause

By default, SNS delivers a message to an SQS subscriber wrapped in a JSON notification envelope with metadata (topic ARN, message id, timestamp, signature) and the original message as an escaped string in the Message field. Message attributes are also moved into the envelope. The consumer was written for messages sent directly to SQS and did not expect the envelope.

Why SNS wraps messages

The envelope carries information some subscribers need: which topic the message came from, a signature to verify authenticity (important for HTTP endpoints), and the original attributes. For SQS subscribers inside your own account, most of that is unnecessary.

Raw message delivery

Setting the subscription attribute RawMessageDelivery to true makes SNS deliver the original message body unchanged to SQS (and to HTTP/S endpoints and Firehose). SNS message attributes become SQS message attributes, up to the SQS limit of 10 attributes per message.

Who is affected

Any consumer moved from a direct SQS producer to an SNS fan-out, consumers written by one team for queues another team later subscribes to SNS, and tests that build SQS events by hand with the bare payload. Lambda consumers using a framework that parses the body automatically fail in the same way, just with a different error, because the parsed object is the envelope. Subscriptions to the same topic can have different settings, so one queue may receive raw messages while another receives envelopes.

Environment

Enable raw message delivery on our subscription

aws sns set-subscription-attributes \
  --subscription-arn arn:aws:sns:eu-west-1:111122223333:order-events:4b1f... \
  --attribute-name RawMessageDelivery --attribute-value true

In Terraform:

resource "aws_sns_topic_subscription" "reporting" {
  topic_arn            = aws_sns_topic.order_events.arn
  protocol             = "sqs"
  endpoint             = aws_sqs_queue.reporting.arn
  raw_message_delivery = true
  filter_policy        = jsonencode({ eventType = ["OrderPlaced", "OrderRefunded"] })
}

After the change, SQS bodies were the original JSON again, and eventType appeared in the record's messageAttributes:

"messageAttributes": {
  "eventType": { "stringValue": "OrderPlaced", "dataType": "String" }
}

Make the consumer tolerant of both shapes

Messages already in the queue still had the envelope, and other teams might subscribe without raw delivery. We added a small unwrapping step so the handler works either way:

import json

def parse_event(record: dict) -> dict:
    body = json.loads(record["body"])
    if isinstance(body, dict) and body.get("Type") == "Notification" and "Message" in body:
        return json.loads(body["Message"])
    return body

def handler(event, context):
    failures = []
    for record in event["Records"]:
        try:
            order = parse_event(record)
            update_reporting(order["orderId"], order["total"], order["currency"])
        except Exception:
            failures.append({"itemIdentifier": record["messageId"]})
    return {"batchItemFailures": failures}

We also enabled partial batch responses (ReportBatchItemFailures) on the event source mapping, so one bad message no longer fails the whole batch of ten.

Filter policies and raw delivery

Our filter policy used message attributes, which works with both modes. SNS can also filter on the message body with FilterPolicyScope: MessageBody, which avoids relying on attributes at all. Either way, the filter is evaluated by SNS before delivery, so it does not care about the delivery format.

Alternatives and trade-offs

• Keep the envelope and always unwrap. Valid if consumers need the topic ARN or SNS message id, for example for deduplication across topics.
• EventBridge instead of SNS for routing by content: richer matching and schemas, with a different envelope (detail) that consumers must handle anyway.

Rolling out the change safely

Switching an existing subscription to raw delivery changes the shape of every new message at once, while messages already in the queue keep the old shape. That is why the tolerant parser shipped first, then the subscription change. Reversing the order would have broken the consumer a second time for the messages in flight.

How to test the fix

aws sns publish --topic-arn arn:aws:sns:eu-west-1:111122223333:order-events \
  --message '{"orderId":"test-1","total":1,"currency":"EUR"}' \
  --message-attributes '{"eventType":{"DataType":"String","StringValue":"OrderPlaced"}}'
aws sqs receive-message --queue-url "$REPORTING_QUEUE_URL" --message-attribute-names All

A unit test covers parse_event with both an enveloped and a raw body.

Confirming in production

After enabling raw delivery and deploying the tolerant consumer, the error rate went to zero, and the messages that had failed during the transition were redriven from the queue's DLQ. Reporting totals matched the order database for the affected period.

Regression guard

• A contract test publishes a sample event to the topic in a test account and asserts the consumer's parsing result.
• Subscription settings, including raw_message_delivery, are managed only in Terraform.
• An alarm on DLQ depth for every consumer queue.

Security implications

Raw delivery drops the SNS signature from the message. For SQS subscribers that is acceptable, because the queue policy controls who can send to the queue; restrict it to the SNS topic with an aws:SourceArn condition. For HTTP/S subscribers, keep the envelope and verify signatures, or anyone who discovers the endpoint can forge messages.

Prevention checklist

• Decide raw versus enveloped delivery explicitly for every SNS subscription.
• Write consumers that recognise and unwrap the SNS envelope if it can appear.
• Remember that message attributes move into the envelope without raw delivery.
• Enable partial batch responses for SQS-triggered Lambda functions.
• Restrict queue policies to the topic ARN.

References

• Amazon SNS raw message delivery: https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html
• Amazon SNS message filtering: https://docs.aws.amazon.com/sns/latest/dg/sns-message-filtering.html
• Fanout to Amazon SQS queues: https://docs.aws.amazon.com/sns/latest/dg/sns-sqs-as-subscriber.html
• Lambda, reporting batch item failures for SQS: https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-errorhandling.html