Tag See all →

#vpc

3 stories

Banner: users could not log in because a Cognito post-authentication Lambda exceeded the 5-second limit

Cloud & Infrastructure Bug Report

Users randomly could not log in: our Cognito post-authentication Lambda exceeded the 5-second limit

Tomás Herrera · Jul 23, 2026 · 7 min read

A few percent of logins failed with a generic error after we added a Cognito post-authentication trigger that recorded login events in our database. The Lambda ran in a VPC, opened a database connection on cold start, and sometimes exceeded Cognito's 5-second trigger timeout. How we moved the work off the login path.

Banner: an EC2 instance in a private subnet cut off from the NAT gateway because of a missing route

Cloud & Infrastructure Bug Report

EC2 in a new private subnet cannot reach the internet or SSM: missing route table association

John Nguyen · May 27, 2026 · 10 min read

Instances in a freshly added private subnet timed out on dnf and never showed up in Session Manager, while identical instances in older subnets were fine. The subnet had silently fallen back to the VPC main route table, which had no NAT route.

Banner: AWS charges for public IPv4 addresses adding up across nodes and idle Elastic IPs

Networking & DNS Bug Report

New AWS line item for public IPv4 addresses: where 2,400 USD a month came from

Ryan Pham · Dec 18, 2024 · 6 min read

A VPC line item we had never paid attention to reached 2,400 USD a month. Since February 2024, AWS charges for every public IPv4 address, in use or idle. Our EKS nodes, idle Elastic IPs and load balancers in many subnets added up. Inventory and cleanup cut it by 70 percent.