API gateway kept routing to dead instances: Eureka self-preservation and stale caches

On a Monday morning our cloud provider reclaimed four spot nodes in one availability zone within a minute. Kubernetes rescheduled the affected pods within about 90 seconds, which is what we expect. What we did not expect was that the API gateway kept sending a quarter of requests for the orders and pricing services to the old pod IPs for six and a half minutes. Clients saw 503s and slow timeouts the whole time.

The gateway logs were full of:

io.netty.channel.AbstractChannel$AnnotatedConnectException: connect timed out: /10.42.7.19:8080
503 SERVICE_UNAVAILABLE "Unable to find instance for orders-service"

And the Eureka dashboard showed the banner:

EMERGENCY! EUREKA MAY BE INCORRECTLY CLAIMING INSTANCES ARE UP WHEN THEY'RE NOT. RENEWALS ARE LESSER THAN THRESHOLD AND HENCE THE INSTANCES ARE NOT BEING EXPIRED JUST TO BE SAFE.

The dead instances were still listed as UP.

Reproducing it

In staging we ran 12 instances of a test service registered with Eureka, then force-deleted four pods with no grace period so no shutdown hook could deregister them. The gateway routed to the dead addresses for between 4 and 7 minutes across runs. Killing only one instance cleared in about 3 minutes. Stopping instances gracefully cleared in under 30 seconds.

What we ruled out

• Kubernetes Service endpoints. The gateway does not use Kubernetes Services for these routes; it resolves instances from Eureka, so endpoint controller timing was irrelevant.
• DNS caching. Routes use lb://orders-service, resolved by Spring Cloud LoadBalancer from the Eureka registry, not DNS.
• A gateway bug. The gateway was faithfully using the instance list it had. The list itself was wrong.

Root cause

Two things stacked. First, the instances died abruptly, so they never sent a cancel to Eureka, and the registry had to notice through missed lease renewals. Second, losing four of twelve instances at once dropped renewals below Eureka's self-preservation threshold (85 percent of expected), so Eureka stopped evicting anything. Instances only disappeared once the replacement pods registered and renewals climbed back over the threshold. On top of that, three caches each held the stale list for up to 30 seconds or more: the Eureka server response cache, the gateway's Eureka client registry fetch, and the Spring Cloud LoadBalancer instance cache.

What is happening under the hood

Eureka is an AP registry built around leases. Each client renews its lease with a heartbeat every 30 seconds by default. If the server sees no renewal within the lease expiration duration (90 seconds by default), an eviction task, running every 60 seconds, removes the instance. A long-standing quirk in Eureka's lease code means an instance can in practice take up to twice the expiration duration to expire.

Self-preservation exists for network partitions. If the Eureka server suddenly stops receiving many renewals, it is more likely that the server is cut off than that most of the fleet died. So when renewals in the last minute fall below 85 percent of the expected rate, it freezes evictions and keeps serving the last known list. That is a sensible default for big fleets; in a small cluster, losing a handful of nodes trips it easily.

The path from "instance died" to "gateway stops routing to it" crosses several timers:

Worst case without self-preservation is several minutes. With self-preservation engaged, it is unbounded until renewals recover.

Environment

Minimal reproduction

# gateway application.yml (before)
spring:
  cloud:
    gateway:
      routes:
        - id: orders
          uri: lb://orders-service
          predicates:
            - Path=/api/orders/**
eureka:
  client:
    service-url:
      defaultZone: http://eureka-0.eureka:8761/eureka/,http://eureka-1.eureka:8761/eureka/

# kill a third of instances without letting them deregister
kubectl get pod -l app=orders -o name | head -n 4 \
  | xargs kubectl delete --grace-period=0 --force
watch -n 2 'curl -s http://eureka-0.eureka:8761/eureka/apps/ORDERS-SERVICE | grep -c "<status>UP"'

Who is affected

Any client-side discovery setup with heartbeat leases and caches: Eureka, Consul with long TTL checks, or a homegrown registry. Most at risk are small clusters, spot or preemptible capacity, and services that do not deregister on shutdown.

We attacked it from four directions: deregister when we can, detect faster when we cannot, shrink caches, and let the gateway route around failures it observes.

1. Deregister on every graceful shutdown

Spring Boot's graceful shutdown plus the Eureka client deregister the instance when the context closes. We made sure pods actually get there:

# service application.yml
server:
  shutdown: graceful
spring:
  lifecycle:
    timeout-per-shutdown-phase: 20s

# deployment.yaml (excerpt)
terminationGracePeriodSeconds: 40
lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "curl -s -X POST localhost:8081/actuator/service-registry -H 'Content-Type: application/json' -d '{\"status\":\"OUT_OF_SERVICE\"}'; sleep 15"]

The preStop marks the instance out of service and waits so gateways pick up the change before the app stops. We also enabled the spot node termination handler so reclaimed nodes are drained gracefully when the two-minute notice arrives.

2. Shorter leases, and self-preservation sized for our cluster

# services
eureka:
  instance:
    lease-renewal-interval-in-seconds: 10
    lease-expiration-duration-in-seconds: 30

# eureka server
eureka:
  server:
    expected-client-renewal-interval-seconds: 10
    eviction-interval-timer-in-ms: 5000
    renewal-percent-threshold: 0.49
    response-cache-update-interval-ms: 5000

expected-client-renewal-interval-seconds must match the new client interval, or the self-preservation threshold is computed against the wrong expected rate. We lowered renewal-percent-threshold so losing a zone's worth of instances (about a third) no longer freezes eviction, while a real partition of the Eureka server from most clients still does. Some teams disable self-preservation entirely in small clusters; we kept it, tuned, because we have seen genuine partitions.

3. Shrink client-side caches

# gateway
eureka:
  client:
    registry-fetch-interval-seconds: 5
spring:
  cloud:
    loadbalancer:
      cache:
        ttl: 5s

4. Route around failures the gateway sees

Even with fast discovery there is a window. We added a retry on connection failures for idempotent methods, so a request that hits a dead instance tries another:

spring:
  cloud:
    gateway:
      routes:
        - id: orders
          uri: lb://orders-service
          predicates:
            - Path=/api/orders/**
          filters:
            - name: Retry
              args:
                retries: 2
                methods: GET,HEAD
                series: SERVER_ERROR
                exceptions: java.net.ConnectException,io.netty.channel.ConnectTimeoutException
      httpclient:
        connect-timeout: 1000

The connect timeout was 30 seconds before; that alone was why clients saw slow failures rather than fast ones.

Alternatives and trade-offs

• Server-side discovery. Routing through Kubernetes Services or a service mesh moves health tracking into the platform, which reacts to readiness probes within seconds. We plan to move there; this fix bought time.
• Health-check based load balancing. spring.cloud.loadbalancer.configurations: health-check makes the gateway probe instances itself. It adds probe traffic per gateway instance.
• Lower intervals mean more heartbeats. At 10 seconds, 300 instances send 30 renewals per second to Eureka: trivial for us, worth checking for large fleets.

How to test the fix

We automated the reproduction as a chaos job in staging:

./chaos/kill-instances.sh orders-service 4
./chaos/measure-stale-routing.sh orders-service --duration 300

The second script sends 50 requests per second through the gateway and records the time until no request targets a killed IP.

Confirming in production

Three weeks later another spot reclaim removed five nodes. Stale routing, measured from gateway access logs as requests sent to IPs that no longer had a running pod, lasted 38 seconds, and the retry filter turned almost all of those into successful responses. Gateway 5xx rate peaked at 0.4 percent for under a minute, against 24 percent in the original incident. The Eureka dashboard did not enter self-preservation.

kubectl logs deploy/gateway --since=10m | grep -c AnnotatedConnectException
curl -s http://eureka-0.eureka:8761/eureka/apps | grep -c "<status>UP"

Regression guard

• The chaos job runs weekly in staging and fails if stale routing exceeds 60 seconds.
• An alert fires on the Eureka self-preservation metric and on gateway connect exceptions above 1 per second.
• A CI check fails service builds missing server.shutdown: graceful or the preStop hook.

Security implications

None beyond availability. Routing to dead IPs is a denial-of-service risk, and in clusters that reuse pod IPs quickly a stale registry entry can point at a different, unrelated pod. Our services require mutual TLS with per-service identities, so a reused IP would fail the handshake rather than receive another service's traffic. Retries are limited to idempotent methods to avoid duplicate writes. The actuator service-registry endpoint is bound to the management port and not exposed through the gateway.

Prevention checklist

• Deregister on shutdown and drain nodes on spot reclaim notices.
• Write down every discovery timer end to end and add them up.
• Size self-preservation thresholds for the failures your cluster actually has.
• Use short connect timeouts and retry idempotent requests on connect failures.

References

• Spring Cloud Netflix docs, Eureka server and client configuration: https://docs.spring.io/spring-cloud-netflix/reference/
• Spring Cloud Commons docs, Spring Cloud LoadBalancer caching and health checks: https://docs.spring.io/spring-cloud-commons/reference/spring-cloud-commons/loadbalancer.html
• Spring Cloud Gateway docs, Retry filter: https://docs.spring.io/spring-cloud-gateway/reference/spring-cloud-gateway/gatewayfilter-factories/retry-factory.html
• Spring Boot docs, Graceful shutdown: https://docs.spring.io/spring-boot/reference/web/graceful-shutdown.html
• Netflix Eureka wiki, Server self-preservation mode: https://github.com/Netflix/eureka/wiki