VPC A peers with B, B peers with C, A cannot reach C: AWS peering is not transitive

A new analytics job running in our analytics VPC needed read access to a reporting replica in the payments VPC. The networking setup looked complete on the diagram:

analytics (10.30.0.0/16) <--peering--> shared-services (10.10.0.0/16) <--peering--> payments (10.40.0.0/16)

The engineer added a route in the analytics VPC sending 10.40.0.0/16 to the peering connection with shared services, because that was "the way to payments". The connection timed out:

$ psql "host=10.40.12.31 port=5432 dbname=reporting user=analytics_ro sslmode=require"
psql: error: connection to server at "10.40.12.31", port 5432 failed: Connection timed out
	Is the server running on that host and accepting TCP/IP connections?

From an instance in shared-services, the same database answered immediately.

Checking the obvious first

• Security groups: the database security group allowed 5432 from 10.30.0.0/16.
• Network ACLs: default allow-all on both subnets.
• Route tables: analytics had 10.40.0.0/16 -> pcx-analytics-shared; shared-services had routes to both peers.

Everything looked "right" if you assumed traffic could pass through shared-services.

Reachability Analyzer

AWS Reachability Analyzer evaluates the path configuration without sending traffic. We ran it from the analytics instance's network interface to the database's interface on port 5432:

aws ec2 create-network-insights-path \
  --source eni-0aa11analytics0001 --destination eni-0bb22paymentsdb01 \
  --protocol tcp --destination-port 5432
aws ec2 start-network-insights-analysis --network-insights-path-id nip-0c1d2e3f4a5b6c7d8

The result was "Not reachable", with the explanation pointing at the peering connection: the destination was outside the CIDR of the VPC on the other side of pcx-analytics-shared, so the peering connection would not carry it.

What we ruled out

• Database listening only locally. It accepted connections from shared-services.
• DNS. We connected by IP.
• Overlapping CIDRs. All three ranges were distinct, which matters for the fix.

Root cause

VPC peering connections are not transitive. A peering connection only carries traffic whose source is in one of the two peered VPCs and whose destination is in the other. Traffic from analytics to payments arrived at the peering connection with shared-services and was dropped, because its destination was not in the shared-services CIDR. There is no setting that changes this.

What peering does and does not do

AWS documents these as unsupported configurations. The design is deliberate: a peering is a point-to-point relationship, not a router.

Why the extra route made it look possible

Route tables accept a route to any CIDR via a peering connection, so adding 10.40.0.0/16 -> pcx-analytics-shared succeeded. The route only means "send it to the peering connection"; the peering connection still drops packets outside the peer's CIDR. That route was misleading and we removed it.

Environment

There are three real options.

Option 1: peer A and C directly

For a single new path, a direct peering between analytics and payments is the fastest fix:

resource "aws_vpc_peering_connection" "analytics_payments" {
  vpc_id        = aws_vpc.analytics.id
  peer_vpc_id   = var.payments_vpc_id
  peer_owner_id = var.payments_account_id
  tags          = { Name = "analytics-payments" }
}

resource "aws_route" "analytics_to_payments" {
  route_table_id            = aws_route_table.analytics_private.id
  destination_cidr_block    = "10.40.0.0/16"
  vpc_peering_connection_id = aws_vpc_peering_connection.analytics_payments.id
}

The payments account accepts the peering and adds the return route to 10.30.0.0/16. Narrow the routes to specific subnets if only the reporting subnet should be reachable.

The drawback: a full mesh of n VPCs needs n(n-1)/2 peerings and routes in every table. With seven VPCs planned, that is 21 peerings.

Option 2: Transit Gateway

A Transit Gateway acts as a regional router. Each VPC attaches once, and route tables on the gateway decide which attachments can reach which. It is transitive by design, and it can also connect VPNs and Direct Connect.

resource "aws_ec2_transit_gateway" "main" {
  description                     = "core"
  default_route_table_association = "disable"
  default_route_table_propagation = "disable"
}

resource "aws_ec2_transit_gateway_vpc_attachment" "analytics" {
  transit_gateway_id = aws_ec2_transit_gateway.main.id
  vpc_id             = aws_vpc.analytics.id
  subnet_ids         = aws_subnet.analytics_tgw[*].id
}

resource "aws_route" "analytics_to_payments_via_tgw" {
  route_table_id         = aws_route_table.analytics_private.id
  destination_cidr_block = "10.40.0.0/16"
  transit_gateway_id     = aws_ec2_transit_gateway.main.id
}

With separate Transit Gateway route tables for segments (for example prod, analytics, shared), you control which VPCs can talk, rather than every attached VPC reaching every other. Transit Gateway costs per attachment-hour and per GB processed, unlike same-region peering, which has no hourly charge.

Option 3: expose the service instead of the network

If analytics needs one database endpoint, not network access to the whole payments VPC, AWS PrivateLink can publish a service behind a Network Load Balancer in payments and consume it as an interface endpoint in analytics. No routing between the VPCs at all, and the payments team controls exactly what is exposed.

What we chose

We created the direct peering that week to unblock the job, and migrated to Transit Gateway over the next two months, because more VPCs were coming. Sensitive data access from analytics later moved to PrivateLink.

How to test the fix

aws ec2 start-network-insights-analysis --network-insights-path-id nip-0c1d2e3f4a5b6c7d8
aws ec2 describe-network-insights-analyses --network-insights-analysis-ids nia-... \
  --query 'NetworkInsightsAnalyses[0].NetworkPathFound'
# true

Then from the analytics instance:

nc -zv 10.40.12.31 5432
# Connection to 10.40.12.31 5432 port [tcp/postgresql] succeeded!

Confirming in production

The analytics job connected through the direct peering, and later through Transit Gateway after the migration. VPC flow logs on the database subnet showed accepted traffic from 10.30.0.0/16 on port 5432 only from the analytics subnets.

Regression guard

• Reachability Analyzer paths for every required cross-VPC flow are defined in Terraform and run after network changes in CI.
• A policy check rejects routes to a peering connection whose destination is not inside the peer VPC's CIDR, which catches the misleading route pattern.
• A network diagram generated from Terraform outputs replaced the hand-drawn one.

Security implications

The failed design would have been a security risk had it worked: transitive access through shared-services would have connected analytics to every VPC shared-services could reach. Direct peering and segmented Transit Gateway route tables keep access explicit. Security groups on the database still restrict access to the analytics subnets and port 5432, and the analytics role is read-only.

Prevention checklist

• Remember that VPC peering is point-to-point and never routes through a third VPC.
• Use Reachability Analyzer before debugging packets by hand.
• Move to Transit Gateway once a mesh would need more than a handful of peerings.
• Consider PrivateLink when only a service, not a network, needs to be shared.
• Plan non-overlapping CIDRs across all VPCs and accounts from the start.

References

• VPC peering limitations: https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-basics.html
• Unsupported VPC peering configurations: https://docs.aws.amazon.com/vpc/latest/peering/invalid-peering-configurations.html
• AWS Transit Gateway: https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html
• VPC Reachability Analyzer: https://docs.aws.amazon.com/vpc/latest/reachability/what-is-reachability-analyzer.html
• AWS PrivateLink: https://docs.aws.amazon.com/vpc/latest/privatelink/what-is-privatelink.html