Correlation ID in API logs but missing in SQS consumers: propagating it with Lambda Powertools

A customer reported that their refund request "disappeared". Our refund flow is an API Gateway endpoint backed by a Lambda function, which validates the request and sends a message to an SQS queue; a second Lambda function processes the refund with the payment provider.

In CloudWatch Logs Insights, the API side was easy to find. Powertools Logger added a correlation_id from API Gateway's request id to every line:

fields @timestamp, level, message, correlation_id
| filter service = "refunds-api" and correlation_id = "c8f1d3a2-6b7e-4e1f-9a0c-2d5b8e7f1a44"

{"level":"INFO","message":"refund accepted","service":"refunds-api",
 "correlation_id":"c8f1d3a2-6b7e-4e1f-9a0c-2d5b8e7f1a44","refund_id":"rf_20931"}

The same query against the consumer's log group returned nothing. Its log lines had correlation_id: null, so we searched by refund_id instead, and only found the logs because the API had happened to log that id.

{"level":"ERROR","message":"provider rejected refund","service":"refunds-worker",
 "correlation_id":null,"refund_id":"rf_20931","error":"amount_exceeds_capture"}

How the logger was set up

Both functions used the same decorator:

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging import correlation_paths

logger = Logger(service="refunds-worker")

@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
def handler(event, context):
    for record in event["Records"]:
        process(record)

The consumer copied the API's configuration, including correlation_id_path=API_GATEWAY_REST, which points at requestContext.requestId. An SQS event has no such field, so the extracted value was None.

What we ruled out

• Logs not shipped. All lines were present in the log group; only the field was empty.
• A Powertools bug. The library did exactly what it was told: read a path from the event.
• Batching. Even single-record batches had no correlation id.

Root cause

Correlation ids do not cross an asynchronous hop on their own. The API function had the id from its incoming event, but did not include it in the SQS message. The consumer's logger was configured to read a path that exists only in API Gateway events, so it found nothing. Each function logged in isolation.

Where Powertools gets the correlation id

inject_lambda_context(correlation_id_path=...) evaluates a JMESPath expression against the incoming Lambda event and sets the result as correlation_id on every log line for that invocation. Built-in paths exist for API Gateway, ALB, EventBridge and others. For SQS, the interesting value is not in the event envelope but in each record, and a batch can contain records from different requests, so one invocation-wide correlation id is the wrong model.

Propagating context through SQS

The id must travel with the message. SQS supports up to 10 message attributes per message, which keep metadata separate from the business payload:

Who is affected

Any Lambda-based flow with an asynchronous hop: SQS, SNS, EventBridge, Kinesis or Step Functions between functions. Teams that share one logger configuration snippet across all functions are especially likely to have consumers reading a path that does not exist in their events.

Environment

Producer: send the id with the message

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging import correlation_paths
import boto3, json

logger = Logger(service="refunds-api")
sqs = boto3.client("sqs")

@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
def handler(event, context):
    refund = validate(json.loads(event["body"]))
    sqs.send_message(
        QueueUrl=REFUNDS_QUEUE_URL,
        MessageBody=json.dumps(refund),
        MessageAttributes={
            "correlation_id": {"DataType": "String", "StringValue": logger.get_correlation_id()},
        },
    )
    logger.info("refund accepted", refund_id=refund["id"])
    return {"statusCode": 202,
            "headers": {"X-Correlation-Id": logger.get_correlation_id()},
            "body": json.dumps({"refundId": refund["id"]})}

We also return the correlation id to the client in a response header, so support can ask customers for it.

Consumer: set the id per record

In the consumer, the correlation id is set for each record before processing, and cleared afterwards:

from aws_lambda_powertools import Logger
from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, process_partial_response
from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord

logger = Logger(service="refunds-worker")
processor = BatchProcessor(event_type=EventType.SQS)

def record_handler(record: SQSRecord):
    attr = record.message_attributes.get("correlation_id")
    logger.set_correlation_id(attr.string_value if attr else record.message_id)
    try:
        refund = record.json_body
        logger.info("processing refund", refund_id=refund["id"])
        process_refund(refund)
    finally:
        logger.set_correlation_id(None)

@logger.inject_lambda_context
def handler(event, context):
    return process_partial_response(event=event, record_handler=record_handler,
                                    processor=processor, context=context)

Falling back to the SQS message id when no attribute is present keeps every line traceable to a message at least. The batch processor also reports partial failures, so one failing refund does not retry the whole batch.

Search across both functions

CloudWatch Logs Insights can query several log groups at once:

fields @timestamp, service, level, message, refund_id
| filter correlation_id = "c8f1d3a2-6b7e-4e1f-9a0c-2d5b8e7f1a44"
| sort @timestamp asc

Run against both log groups, this returns the full story of a request in order.

Alternatives and trade-offs

• Distributed tracing with X-Ray or OpenTelemetry gives a visual trace across the queue and timings, and Powertools Tracer integrates with it. We use both: traces for timing, correlation ids in logs for text search and retention beyond trace sampling.
• Body field instead of attribute if the message also crosses systems that drop SQS attributes, for example when forwarded through other services.

How to test the fix

cid=$(curl -s -D - -o /dev/null -XPOST https://api.example.com/refunds -d @refund.json \
      | awk -F': ' 'tolower($1)=="x-correlation-id"{print $2}' | tr -d '\r')
sleep 20
aws logs start-query --log-group-names /aws/lambda/refunds-api /aws/lambda/refunds-worker \
  --start-time $(date -d '-10 min' +%s) --end-time $(date +%s) \
  --query-string "fields service, message | filter correlation_id = \"$cid\""

Expected: lines from both services.

Confirming in production

All consumer log lines carried a correlation id after the deploy. The customer's refund was found immediately in the next similar case: the provider rejected it because the amount exceeded the captured amount, which support could explain and fix.

Regression guard

• A CloudWatch metric filter counts consumer log lines with correlation_id null; an alarm fires if more than 1% of lines have none.
• The shared producer helper for SQS always adds the attribute; direct send_message calls are flagged in review.

Security implications

Correlation ids must not contain personal data or secrets; ours are random UUIDs. Returning the id to clients is safe for the same reason. Message attributes are visible to anyone with sqs:ReceiveMessage on the queue, like the body, so they get the same protection.

Prevention checklist

• Propagate correlation ids explicitly across every asynchronous hop.
• Use SQS message attributes for metadata, with a fallback to the message id.
• Set correlation ids per record in batch consumers, not per invocation.
• Do not copy logger configuration between functions with different event sources.
• Alert on log lines without correlation ids.

References

• Powertools for AWS Lambda (Python), Logger: https://docs.powertools.aws.dev/lambda/python/latest/core/logger/
• Powertools for AWS Lambda (Python), Batch processing: https://docs.powertools.aws.dev/lambda/python/latest/utilities/batch/
• Amazon SQS message attributes: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html
• CloudWatch Logs Insights query syntax: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html