A routine database failover lasted 20 seconds, but our API stayed down for 41 minutes. Three layers each made up to three attempts with no delay, multiplying load on the database by 27. Here is the math, the fix and how we proved it.
Our document tagger hit HTTP 429 constantly while sending only 40 requests a minute, far below the request limit. The token-per-minute limit was the real ceiling, inflated by an oversized max_tokens and minute-aligned bursts. How we read the headers and built a client-side limiter.
A routine rolling deploy restarted one realtime pod holding 50,000 WebSocket connections. Every client reconnected within two seconds, the auth and history lookups melted, and the storm kept itself alive for 25 minutes. Here is how we drain sockets, pace reconnects and make resume cheap.