EventBridge rule matched but the SQS queue in another account stayed empty: queue and KMS policies

The analytics team asked for a feed of OrderPlaced and OrderRefunded events. Our platform account already published them to a custom event bus. The plan was a rule on that bus with an SQS queue in the analytics account as its target. We created the rule, the analytics team created the queue, and nothing arrived.

Is the rule matching?

EventBridge publishes per-rule metrics in CloudWatch. For the rule orders-to-analytics:

MatchedEvents        1,842 in the last hour
Invocations          1,842
FailedInvocations    1,842

Every matched event was delivered to the target, and every delivery failed. The console showed no error on the rule itself; FailedInvocations was the only sign.

Why do invocations fail?

EventBridge does not log the reason for each failed delivery by default. Two ways to get it: configure a dead-letter queue on the target, whose messages include error attributes, or check CloudTrail in the target account. We added a DLQ in the platform account first:

aws events put-targets --event-bus-name orders --rule orders-to-analytics --targets '[{
  "Id": "analytics-queue",
  "Arn": "arn:aws:sqs:eu-west-1:444455556666:analytics-order-events",
  "DeadLetterConfig": { "Arn": "arn:aws:sqs:eu-west-1:111122223333:eventbridge-dlq" },
  "RetryPolicy": { "MaximumRetryAttempts": 10, "MaximumEventAgeInSeconds": 3600 }
}]'

Messages in the DLQ carried attributes explaining the failure:

ERROR_CODE     an SQS access-denied code (after the first fix: a KMS access error)
ERROR_MESSAGE  Access to the resource https://sqs.eu-west-1.amazonaws.com/... is denied.
RULE_ARN       arn:aws:events:eu-west-1:111122223333:rule/orders/orders-to-analytics
TARGET_ARN     arn:aws:sqs:eu-west-1:444455556666:analytics-order-events

Problem 1: the queue policy

The analytics queue's access policy allowed our account root:

{
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:eu-west-1:444455556666:analytics-order-events"
}

That grants access to IAM principals in our account. EventBridge does not send messages to SQS as an IAM principal of our account; it calls SQS as the EventBridge service. For SQS targets, the queue policy must allow the service principal events.amazonaws.com, ideally restricted to the specific rule ARN.

Problem 2: the KMS key

After fixing the queue policy, the error changed to a KMS failure. The analytics queue used server-side encryption with a customer managed KMS key. To send a message to an encrypted queue, the sender needs kms:GenerateDataKey and kms:Decrypt on that key. The key policy allowed only principals in the analytics account.

What we ruled out

• Event pattern errors. MatchedEvents proved the pattern was right.
• The default AWS managed key alias/aws/sqs. The queue did not use it; if it had, cross-account delivery would be impossible, because AWS managed key policies cannot be changed to allow other services.
• Network or VPC endpoints. EventBridge to SQS delivery happens between AWS services, not through our VPCs.

Root cause

EventBridge delivers to SQS targets as the EventBridge service principal. The queue policy allowed our account's IAM principals instead of events.amazonaws.com, and the queue's customer managed KMS key did not allow EventBridge to use it. Without a target DLQ, the failures were visible only as a metric.

Two authorization models for EventBridge targets

Cross-account SQS is in the first group: everything depends on the queue policy in the target account. The usual alternative for cross-account delivery is to send events to an event bus in the other account and let that account's own rules deliver to its queue, which keeps each team in control of its own resources.

Why failures are quiet

EventBridge retries failed deliveries with backoff for up to 24 hours by default, then drops the event. Without a DLQ, there is no record of what was lost except the metrics.

Environment

Queue policy in the analytics account

{
  "Sid": "AllowOrdersRuleFromPlatform",
  "Effect": "Allow",
  "Principal": { "Service": "events.amazonaws.com" },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:eu-west-1:444455556666:analytics-order-events",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:events:eu-west-1:111122223333:rule/orders/orders-to-analytics"
    }
  }
}

The aws:SourceArn condition matters: without it, any EventBridge rule in any account could send to the queue.

Key policy in the analytics account

{
  "Sid": "AllowEventBridgeToUseKey",
  "Effect": "Allow",
  "Principal": { "Service": "events.amazonaws.com" },
  "Action": ["kms:GenerateDataKey", "kms:Decrypt"],
  "Resource": "*",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:events:eu-west-1:111122223333:rule/orders/orders-to-analytics"
    }
  }
}

Within seconds of applying both, FailedInvocations dropped to zero and messages arrived in the queue. We then re-sent the events captured in the DLQ by moving them back through a small redrive script that publishes the original event payloads to the bus.

Keep the DLQ and alarms

The DLQ and retry policy stay. We also added alarms:

resource "aws_cloudwatch_metric_alarm" "eventbridge_failed" {
  alarm_name          = "orders-to-analytics-failed-invocations"
  namespace           = "AWS/Events"
  metric_name         = "FailedInvocations"
  dimensions          = { RuleName = "orders-to-analytics", EventBusName = "orders" }
  statistic           = "Sum"
  period              = 300
  evaluation_periods  = 1
  threshold           = 0
  comparison_operator = "GreaterThanThreshold"
  alarm_actions       = [var.oncall_topic_arn]
}

An alarm on the DLQ's ApproximateNumberOfMessagesVisible above zero completes the picture.

Alternatives and trade-offs

• Bus-to-bus delivery: the platform rule targets the analytics account's event bus using an IAM role, and the analytics account routes to its own queue. More hops, but each team owns its policies, and adding consumers in the analytics account does not need changes in ours.
• EventBridge Pipes from a queue in the source account, if the consumer prefers pulling.

How to test the fix

aws events put-events --entries '[{
  "EventBusName": "orders", "Source": "acme.orders", "DetailType": "OrderPlaced",
  "Detail": "{\"orderId\":\"test-123\",\"total\":42.5}"
}]'
aws sqs receive-message --queue-url https://sqs.eu-west-1.amazonaws.com/444455556666/analytics-order-events \
  --profile analytics --wait-time-seconds 10

Confirming in production

Invocations equalled MatchedEvents and FailedInvocations stayed at zero for the next week. The analytics team's consumer confirmed event counts against our order database daily for the first month.

Regression guard

• Every EventBridge target in our Terraform module requires a DLQ and a retry policy.
• Alarms on FailedInvocations per rule and on DLQ depth.
• A cross-account integration test publishes a test event after each deploy of either account's policies.

Security implications

The first instinct during debugging was to allow "Principal": "*" on the queue to "see if it works". That would have let anyone send messages to the analytics pipeline. Service principals with aws:SourceArn conditions grant exactly the access needed. Order events contain customer identifiers, so the target queue's encryption and its consumers' access were reviewed by the data protection team.

Prevention checklist

• For SQS, SNS and Lambda targets, allow events.amazonaws.com in the resource policy with aws:SourceArn.
• Allow EventBridge in the KMS key policy of encrypted targets; never use the AWS managed SQS key cross-account.
• Configure a DLQ and retry policy on every target.
• Alarm on FailedInvocations, not only on downstream processing.
• Prefer bus-to-bus delivery for cross-account integrations owned by different teams.

References

• Amazon EventBridge, Using resource-based policies: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-use-resource-based.html
• Amazon EventBridge, Event retry policy and dead-letter queues: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rule-dlq.html
• Amazon SQS, Encryption at rest and key policies: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-key-management.html
• Amazon EventBridge, Sending events between AWS accounts: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-cross-account.html