AWS Site-to-Site VPN tunnel shows UP but nothing gets through: routes, selectors and MSS

We needed a private connection from a branch office to a VPC so on-premises staff could reach an internal application without exposing it to the internet. We created a virtual private gateway, a customer gateway for the office's public IP and a Site-to-Site VPN connection with static routing, downloaded the generated configuration for the office firewall and applied it.

Within minutes both tunnels showed UP in the AWS console, and the firewall showed both IPsec phase 2 security associations established. From an office laptop:

$ ping 10.50.1.20
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1

10.50.1.20 was an EC2 instance in a private subnet of the VPC. A green tunnel only proves that IKE and IPsec negotiated; it says nothing about whether traffic is routed into it, allowed through, or able to return.

Problem 1: no return route in the VPC

The VPN tunnel metrics showed TunnelDataIn (traffic arriving from the office into AWS) increasing, but TunnelDataOut stayed at zero. Packets reached AWS, and replies never went back. The private subnet's route table:

aws ec2 describe-route-tables --route-table-ids rtb-0a1b2c3d4e5f60718 \
  --query 'RouteTables[0].Routes[].[DestinationCidrBlock,GatewayId,NatGatewayId,Origin]' --output table

| 10.50.0.0/16 | local        | None            | CreateRouteTable |
| 0.0.0.0/0    | None         | nat-0f1e2d3c4b  | CreateRoute      |

No route for the office range 192.168.40.0/22. Replies from the instance followed the default route to the NAT gateway and out to the internet, where they were dropped. We had attached the virtual private gateway to the VPC but not enabled route propagation on the subnet's route table.

aws ec2 enable-vgw-route-propagation --route-table-id rtb-0a1b2c3d4e5f60718 --gateway-id vgw-0123abcd4567ef890

The route 192.168.40.0/22 -> vgw-... appeared with origin EnableVgwRoutePropagation, and the instance's security group needed a rule allowing ICMP and HTTPS from 192.168.40.0/22. Pings worked.

Problem 2: only some office subnets worked

The next day, the finance team on 192.168.42.0/24 still could not connect, while staff on 192.168.40.0/24 could. The firewall log showed traffic from finance being sent to the internet instead of the tunnel. Its phase 2 configuration:

local subnet:  192.168.40.0/24
remote subnet: 10.50.0.0/16

The traffic selector covered only one office subnet, although the AWS customer gateway static route said 192.168.40.0/22. Firewall traffic from 192.168.42.0/24 did not match any IPsec policy and was routed, and NATed, to the internet. We changed the local selector to 192.168.40.0/22 and added the rule that exempts VPN-bound traffic from source NAT. With route-based VPNs (virtual tunnel interfaces) instead of policy-based selectors, the firewall uses routes, which avoids this class of mismatch.

Problem 3: small requests fine, large transfers hang

The application worked for browsing, but uploading files over a few hundred kilobytes stalled, and a git clone over HTTPS froze partway. This pattern, where small packets pass and large ones disappear, is almost always MTU.

ping -M do -s 1372 10.50.1.20   # 1372 + 28 = 1400 bytes, passes
ping -M do -s 1450 10.50.1.20   # 1478 bytes, no reply and no "fragmentation needed"

IPsec adds headers, so the largest packet that fits through the tunnel is smaller than the 1500-byte Ethernet MTU. Hosts rely on ICMP "fragmentation needed" messages to discover the smaller path MTU, and those were being filtered somewhere along the path. TCP connections negotiated a 1460-byte MSS, sent full-size segments, and those segments were dropped silently.

What we ruled out

• Tunnel instability. Both tunnels stayed up throughout; CloudWatch TunnelState was constant.
• Instance firewall. iptables and the OS firewall on the EC2 instance were open for the office range.
• Asymmetric tunnel use. With static routing, AWS can return traffic on either tunnel; the firewall accepted both, which we confirmed in its logs.

Root causes

Three independent misconfigurations: the VPC subnet route table did not have routes to the office (propagation disabled), the firewall's traffic selector covered only part of the office range and the rest was NATed to the internet, and the effective MTU through IPsec was lower than hosts assumed with path MTU discovery broken.

What "UP" means

A tunnel is UP when IKE phase 1 and IPsec phase 2 negotiations succeed. Routing, filtering and packet size are separate layers on top. A useful mental checklist follows the packet in both directions: does the source route it into the tunnel, does the tunnel accept it (selectors), is it allowed at the destination (security groups, NACLs), and does the reply have a route back?

MTU and MSS through IPsec

AWS documents a maximum MTU of 1446 bytes and recommends clamping TCP MSS to 1379 bytes on the customer gateway, which accounts for the worst-case IPsec overhead with the supported algorithms. MSS clamping rewrites the MSS option in TCP handshakes passing through the firewall, so both ends agree on segments small enough to fit, without relying on ICMP.

Environment

The final working configuration had four parts:

1. Route propagation from the virtual private gateway enabled on every private route table that hosts resources the office needs.
2. Security groups allowing the office CIDR to the specific ports needed, not all traffic.
3. Firewall traffic selector covering the whole office range (192.168.40.0/22 to 10.50.0.0/16) and a NAT exemption for that traffic.
4. MSS clamping to 1379 on the firewall's tunnel interface:

# vendor-neutral representation of the firewall setting
set vpn ipsec tunnel aws-tunnel-1 tcp-mss 1379
set vpn ipsec tunnel aws-tunnel-2 tcp-mss 1379

In Terraform, route propagation is one resource per route table:

resource "aws_vpn_gateway_route_propagation" "private" {
  count          = length(aws_route_table.private)
  vpn_gateway_id = aws_vpn_gateway.main.id
  route_table_id = aws_route_table.private[count.index].id
}

Alternatives and trade-offs

• BGP (dynamic routing) instead of static routes advertises office prefixes automatically and handles tunnel failover more cleanly. It needs firewall support for BGP over IPsec.
• Transit Gateway as the VPN endpoint instead of a virtual private gateway, if the office needs to reach several VPCs.
• AWS Client VPN or a zero trust access service if the requirement is user access to an application, not a network-to-network link.

How to test the fix

# from the office, a host in each office subnet
ping -c3 10.50.1.20
ping -M do -s 1372 10.50.1.20      # 1400-byte packet passes
curl -so /dev/null -w '%{http_code} %{size_download}\n' https://app.internal.example.com/files/large.bin

Confirming in production

Both office subnets reached the application; large uploads completed at line speed. TunnelDataIn and TunnelDataOut both showed traffic, and tunnel failover tested by disabling one tunnel on the firewall moved traffic to the other within about 30 seconds.

Regression guard

• CloudWatch alarms on TunnelState for each tunnel and on TunnelDataOut dropping to zero during office hours.
• A synthetic check from an office host fetches a 5 MB file through the tunnel every 15 minutes.
• The firewall configuration and the AWS side are both in version control, and the office CIDR is a single shared variable.

Security implications

The NAT misconfiguration sent finance traffic intended for a private network to the internet instead. The application required TLS, so the content was not exposed, but connection attempts leaked internal hostnames. Security groups were restricted to the office range and needed ports, and the application kept its own authentication: the VPN is a network path, not an identity check.

Prevention checklist

• After a tunnel comes up, verify routes in both directions before testing applications.
• Enable route propagation, or add static routes, on every route table that needs the remote range.
• Keep traffic selectors, customer gateway routes and NAT exemptions aligned on the full CIDR.
• Clamp TCP MSS to 1379 on the customer gateway.
• Monitor tunnel state and data in both directions.

References

• AWS Site-to-Site VPN troubleshooting: https://docs.aws.amazon.com/vpn/latest/s2svpn/Troubleshooting.html
• Customer gateway device requirements, MTU and MSS: https://docs.aws.amazon.com/vpn/latest/s2svpn/your-cgw.html
• Route propagation for a virtual private gateway: https://docs.aws.amazon.com/vpn/latest/s2svpn/VPNRoutingTypes.html
• Monitoring VPN tunnels with CloudWatch: https://docs.aws.amazon.com/vpn/latest/s2svpn/monitoring-cloudwatch-vpn.html