EC2 in a new private subnet cannot reach the internet or SSM: missing route table association

We added a third Availability Zone to a production VPC so an Auto Scaling group could spread across eu-west-1a, 1b and 1c. The first instances launched into the new 1c private subnet never became healthy. User data failed on the first package install, and the instances never appeared in Systems Manager, so we could not even open a Session Manager shell to look. Instances from the same launch template in 1a and 1b were fine.

The cloud-init log, fetched with aws ec2 get-console-output, showed the package step timing out:

Error: Failed to download metadata for repo 'amazonlinux': Cannot download repomd.xml: Curl error (28): Timeout was reached for https://al2023-repos-eu-west-1-xxxxxxxx.s3.dualstack.eu-west-1.amazonaws.com/core/mirrors/2023.7.20250512/x86_64/mirror.list [Connection timed out after 30001 milliseconds]

When we later got onto one of the instances through EC2 Serial Console, the SSM Agent log told the same story:

ERROR [HealthCheck] error when calling AWS APIs. error details - RequestError: send request failed
caused by: Post "https://ssm.eu-west-1.amazonaws.com/": dial tcp 198.51.100.24:443: i/o timeout

Reproducing it

Launch any instance into the new subnet and try an outbound HTTPS call. Every outbound connection to a public address times out; traffic inside the VPC CIDR works.

curl -sS -m 5 https://checkip.amazonaws.com || echo "no egress"
# curl: (28) Connection timed out after 5001 milliseconds
# no egress

What we ruled out

• Security group egress. The instance security group was the same one used in the healthy AZs and allowed all outbound traffic.
• Network ACL. The new subnet used the VPC default NACL, which allows everything in both directions. We checked anyway because a custom NACL that forgets the ephemeral return ports (1024-65535) produces exactly this kind of timeout.
• DNS. dig ssm.eu-west-1.amazonaws.com resolved normally through the VPC resolver, so enableDnsSupport was on and this was not a name resolution failure. The timeout was on the TCP connect.
• The NAT gateway itself. Instances in 1a were egressing through it, so it was up and its public subnet routed to the internet gateway.

Root cause

We asked AWS which route table the subnet used:

aws ec2 describe-route-tables \
  --filters Name=association.subnet-id,Values=subnet-0c1c0c1c0c1c0c1c0 \
  --query 'RouteTables[].RouteTableId'
[]

An empty list means the subnet has no explicit association, so it implicitly uses the VPC's main route table. Our main route table contained only the local route. The private route tables that carry 0.0.0.0/0 -> nat-... were associated with the 1a and 1b subnets by name in Terraform. The change that added 1c created the subnet but not the aws_route_table_association, and AWS does not complain about that: a subnet without an association is perfectly valid, it just gets the main table.

What is happening under the hood

Every subnet in a VPC is associated with exactly one route table. If you associate one explicitly, that table is used. If you do not, the subnet uses the main route table, which AWS creates with the VPC. The console shows this as "Main: Yes" on the table and an "implicit" association on the subnet, but neither the API response for subnet creation nor Terraform plan output warns you.

For a private subnet to reach public endpoints such as package repositories or the Systems Manager API, its route table needs a default route to a NAT gateway (or another egress path), and the NAT gateway must sit in a public subnet whose route table points 0.0.0.0/0 at an internet gateway. Break any link and outbound TCP simply hangs, because packets leave the instance and have nowhere to go. There is no ICMP error to make the failure fast.

SSM Agent makes this worse to diagnose: it needs outbound HTTPS to the ssm, ssmmessages and ec2messages endpoints to register, so the tool you would normally use to debug a broken instance is the first thing to fail.

Environment

Minimal reproduction

The broken Terraform looked roughly like this. The for_each on the associations was driven by a separate, hand-maintained map that nobody updated:

resource "aws_subnet" "private" {
  for_each          = var.private_subnets   # now includes "c"
  vpc_id            = aws_vpc.main.id
  cidr_block        = each.value.cidr
  availability_zone = each.value.az
}

resource "aws_route_table_association" "private" {
  for_each       = var.private_route_assoc  # still only "a" and "b"
  subnet_id      = aws_subnet.private[each.key].id
  route_table_id = aws_route_table.private[each.key].id
}

Who is affected

Any VPC where subnets are added in a separate change from their route table associations: hand-written Terraform, CloudFormation templates where the association is a separate resource, and console work. Teams using a module that creates subnet, route table and association together, keyed by the same map, are usually safe.

The fix

First the immediate repair, done with the CLI so the new AZ could serve traffic while the code change went through review:

aws ec2 associate-route-table \
  --subnet-id subnet-0c1c0c1c0c1c0c1c0 \
  --route-table-id rtb-0priv1c0000000000

Then the real fix: drive subnets and associations from the same key so one cannot exist without the other.

 resource "aws_route_table_association" "private" {
-  for_each       = var.private_route_assoc
+  for_each       = aws_subnet.private
   subnet_id      = aws_subnet.private[each.key].id
   route_table_id = aws_route_table.private[each.key].id
 }

After the change, terraform plan wanted to create the association for c, which already existed from the CLI repair, so we imported it instead of letting the apply fail on a duplicate:

terraform import 'aws_route_table_association.private["c"]' subnet-0c1c0c1c0c1c0c1c0/rtb-0priv1c0000000000

We also made the main route table useless as a fallback on purpose. It now has no routes beyond local, and we manage it explicitly so nobody adds a default route to it later:

resource "aws_default_route_table" "main" {
  default_route_table_id = aws_vpc.main.default_route_table_id
  tags = { Name = "main-do-not-use" }
}

A subnet that misses its association now fails loudly and in the same way every time, instead of accidentally inheriting whatever routes someone put on the main table.

Alternatives and trade-offs

• VPC endpoints for SSM. Interface endpoints for ssm, ssmmessages and ec2messages, with private DNS enabled and a security group allowing 443 from the VPC CIDR, let Session Manager work with no internet path at all. We added them afterwards so we can always get a shell, even when egress is broken. They cost per endpoint per AZ, so weigh that against the number of VPCs.
• S3 gateway endpoint. Amazon Linux 2023 repositories are served from S3, so a free S3 gateway endpoint attached to the private route tables lets dnf work without NAT. Gateway endpoints are added as routes, so they also depend on the correct route table association.
• One shared private route table. Fewer tables means fewer associations to forget, but you lose per-AZ NAT routing and an AZ failure in the NAT's zone takes egress down everywhere.

How to test the fix

From an instance in the repaired subnet:

curl -sS -m 5 https://checkip.amazonaws.com
# 203.0.113.57   <- the Elastic IP of the NAT gateway in eu-west-1c
sudo dnf -q makecache && echo "repos ok"

The returned address should match the NAT gateway in the same AZ. If it matches another AZ's NAT, the subnet is associated with the wrong private table.

We also ran VPC Reachability Analyzer from the instance to the internet gateway, which explains a failure hop by hop instead of timing out:

path_id=$(aws ec2 create-network-insights-path \
  --source i-0abc1c0000000000 --destination igw-0123456789abcdef0 \
  --protocol tcp --destination-port 443 \
  --query 'NetworkInsightsPath.NetworkInsightsPathId' --output text)
aws ec2 start-network-insights-analysis --network-insights-path-id "$path_id"

Before the fix, the analysis reported no route to the destination in the main route table. After, it returned NetworkPathFound: true.

Confirming in production

Systems Manager is the quickest fleet-wide signal, because every instance with working egress registers within a couple of minutes:

aws ssm describe-instance-information \
  --query 'InstanceInformationList[].[InstanceId,PingStatus]' --output table

All instances in 1c showed Online. We also watched the NAT gateway metrics: ActiveConnectionCount on the 1c NAT went from zero to a level similar to the other two AZs, and ErrorPortAllocation stayed at zero.

Regression guard

We added a check to CI that fails when any subnet in the account has no explicit association. It runs against the real account after terraform apply in staging:

#!/usr/bin/env bash
set -euo pipefail
vpc_id="$1"
all=$(aws ec2 describe-subnets --filters Name=vpc-id,Values="$vpc_id" \
  --query 'Subnets[].SubnetId' --output text | tr '\t' '\n' | sort)
assoc=$(aws ec2 describe-route-tables --filters Name=vpc-id,Values="$vpc_id" \
  --query 'RouteTables[].Associations[?SubnetId].SubnetId' --output text | tr '\t' '\n' | sort)
orphans=$(comm -23 <(echo "$all") <(echo "$assoc"))
if [ -n "$orphans" ]; then
  echo "Subnets using the main route table implicitly:"; echo "$orphans"; exit 1
fi

Security implications

The bug itself reduced exposure rather than increasing it: the instances had no egress at all. The risk is in the fallback. If someone had once added a route to an internet gateway on the main table, every subnet that missed its association would have become a public subnet without anyone deciding so. Keeping the main table route-free removes that path. The SSM interface endpoints we added keep shell access inside the VPC and let us keep SSH closed entirely; we restricted their security group to the VPC CIDR.

Prevention checklist

• Create subnet, route table and association in one module, keyed by the same map.
• Keep the main route table empty apart from local and manage it in code.
• Run an orphan-subnet check after every network change.
• Add SSM interface endpoints so you can debug instances even when egress fails.
• Test new subnets with an outbound call that shows the egress IP, not only with a ping.

References

• Route tables and subnet associations: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html
• NAT gateways: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html
• Systems Manager VPC endpoints: https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-create-vpc.html
• VPC Reachability Analyzer: https://docs.aws.amazon.com/vpc/latest/reachability/what-is-reachability-analyzer.html
• Terraform aws_default_route_table: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/default_route_table