Route 53 failover not triggering while the primary region was down

At 14:06 UTC our order API in the primary region started failing every write. Reads mostly worked. The on-call engineer saw error rates climb to 62% on POST endpoints and waited for DNS failover to the secondary region, which we had built and tested in a game day four months earlier. It never happened. Route 53 kept answering api.example.com with the primary load balancer for the full 38 minutes until we failed over by hand.

The application logs in the primary region were full of connection errors to the database writer:

Error: connect ETIMEDOUT
    at PoolConnection._handleTimeoutError (node_modules/mysql2/lib/connection.js:212:17)
code: 'ETIMEDOUT', fatal: true

The Route 53 console showed the primary health check as Healthy throughout.

Forty minutes after the region recovered we had a second, shorter outage. Route 53 automatically moved traffic back to the primary as soon as its health check passed, but we had already promoted the secondary database during the manual failover. The primary region was now pointing at a read-only replica:

Error: The MySQL server is running with the --read-only option so it cannot execute this statement
code: 'ER_OPTION_PREVENTS_STATEMENT', errno: 1290, sqlState: 'HY000'

So we had two bugs: failover did not trigger when it should, and failback triggered when it should not.

Reproducing it

We rebuilt the setup in a sandbox account and reproduced the first bug in ten minutes:

1. Deploy the API to both regions behind an ALB each, with failover CNAME records and a Route 53 HTTPS health check on the primary.
2. Add a security group rule in the primary region that drops traffic from the app tasks to the Aurora writer endpoint.
3. Watch POST /orders fail, and watch the health check stay green.

aws route53 get-health-check-status \
  --health-check-id 1a2b3c4d-1111-2222-3333-444455556666 \
  --query 'HealthCheckObservations[].StatusReport.Status' --output text

Success: HTTP Status Code 200, OK. Resolved IP: 203.0.113.24
Success: HTTP Status Code 200, OK. Resolved IP: 203.0.113.24
Success: HTTP Status Code 200, OK. Resolved IP: 198.51.100.17

What we ruled out

• Health checkers blocked by a security group. If Route 53 could not reach the endpoint, the check would go unhealthy, which is the opposite of what we saw. The checker IP ranges (ROUTE53_HEALTHCHECKS in the published AWS IP ranges) were allowed.
• Wrong record attached to the health check. aws route53 list-resource-record-sets showed the PRIMARY record with the expected HealthCheckId, and the SECONDARY record pointing at the other region.
• Route 53 itself being impaired. Route 53's data plane, which answers queries and evaluates health checks, is designed to keep working through regional events. Other zones in our account failed over correctly in the same window during a test.
• Client DNS caching alone. Caching makes failover slow, but here the authoritative answer itself never changed, which we confirmed with test-dns-answer.

aws route53 test-dns-answer --hosted-zone-id Z0123456789EXAMPLE \
  --record-name api.example.com --record-type CNAME

RecordData: api-primary-123456789.us-east-1.elb.amazonaws.com
ResponseCode: NOERROR

Root cause

The health check requested /health, and /health was a liveness endpoint: it returned 200 if the Node.js process was running and could respond. It did not touch the database. The load balancer, the containers and the process were all fine. The thing that was broken was the path from the containers to the database writer, and nothing in the health check exercised it. Route 53 did exactly what it was configured to do.

The failback problem had the same shape from the other side. The primary record's health would go green the moment the process answered, and Route 53 failover routing always prefers a healthy primary. Nothing in the design said "the primary is only eligible if it can actually write to the current database writer".

What is happening under the hood

Route 53 failover routing is simple: for a query on a record set with PRIMARY and SECONDARY records, return the primary if its associated health check is healthy, otherwise return the secondary. Health checkers in multiple locations probe the endpoint at the configured interval (10 or 30 seconds), and a check is marked unhealthy after FailureThreshold consecutive failures as reported by enough checkers. Route 53 decides health from the status code (2xx or 3xx is healthy) and optionally a string match in the first 5,120 bytes of the body.

That means the health check only knows what the endpoint tells it. A shallow endpoint turns a regional dependency failure into a green light.

Two more behaviours matter here:

• Fail-open when everything is unhealthy. If both primary and secondary are unhealthy, Route 53 treats the records as healthy rather than returning nothing. A deep health check that includes a shared dependency (for example a global third-party API) could therefore mark both regions unhealthy at once and achieve nothing.
• Automatic failback. Failover routing has no memory. As soon as the primary is healthy again, answers switch back. There is no built-in "stay on secondary until a human says so".

Finally, DNS answers live in caches. Our failover records had TTL 300, and our Java clients used the JVM's own DNS cache, so even a correct failover would have taken several minutes to reach everyone.

Environment

Minimal reproduction

resource "aws_route53_health_check" "primary" {
  fqdn              = "api-primary.example.com"
  type              = "HTTPS"
  port              = 443
  resource_path     = "/health"     # liveness only
  failure_threshold = 3
  request_interval  = 30
}

resource "aws_route53_record" "api_primary" {
  zone_id         = var.zone_id
  name            = "api.example.com"
  type            = "CNAME"
  ttl             = 300
  records         = [var.primary_alb_dns]
  failover_routing_policy { type = "PRIMARY" }
  health_check_id = aws_route53_health_check.primary.id
  set_identifier  = "primary"
}

// The endpoint that hid the outage
app.get('/health', (req, res) => res.status(200).send('ok'));

Who is affected

Any failover design where the health check endpoint does not exercise the dependency whose loss should trigger failover. It is especially common when the same /health path is shared by the ALB target group (which should be shallow, so one slow database does not drain every target) and by Route 53 (which needs to be deep). Anyone using failover routing with a database that has a single writer is also exposed to the automatic failback problem.