WebSocket reconnect storm: one pod restart dropped 50K sockets and the cluster

At 14:02 on a Thursday we shipped a one-line change to the realtime gateway, the Node.js service that holds WebSocket connections for our team-chat and notifications product. The rollout replaced pods one at a time. When the first old pod terminated, error rates across the whole product went to 40 percent and stayed there for 25 minutes, long after the rollout had finished.

The first alerts came from the ingress layer:

2024-11-14 14:02:19 [alert] 31#31: 4096 worker_connections are not enough while connecting to upstream
2024-11-14 14:02:21 [error] 29#29: *8812201 upstream timed out (110: Connection timed out) while reading response header from upstream

Then from the gateway pods themselves:

Error: accept EMFILE
    at TCP.onconnection (node:net:2122:24)

And finally from the session store: PostgreSQL CPU at 100 percent, with the top query being the "messages since cursor" lookup that clients run after every reconnect.

Reproducing it

In a staging cluster with six gateway pods we opened 60,000 connections with a load script that mimicked our real client: on close, wait one second, reconnect, authenticate, then fetch everything since the last message id. Deleting one pod with kubectl delete pod reproduced the outage every time. With the reconnect delay randomised, the same test produced a brief bump and no errors.

What we ruled out

• The code change. Rolling back did nothing; the storm continued because it no longer depended on which version was running.
• Pod resources. Gateway pods had headroom for 15,000 steady connections each. The problem was the rate of new connections, not the count.
• The load balancer algorithm alone. Ingress used round robin, so the 50,000 returning clients were spread fairly. Fair spreading of a 50,000 handshake spike still overwhelmed every pod.

Root cause

One pod held about 50,000 connections. When Kubernetes sent SIGTERM, our process exited almost immediately, closing all of them at the same instant. Every client used a fixed one second reconnect delay, so 50,000 TLS handshakes, JWT verifications, presence writes and history queries arrived inside roughly two seconds. The history query hit PostgreSQL hard, handshakes queued, clients timed out after five seconds, gave up, waited one second and tried again. Failed attempts multiplied the load, so the remaining pods began dropping their own healthy connections under memory pressure, which added more clients to the storm. The system had a stable bad state and no mechanism to leave it.

What is happening under the hood

A long-lived connection is cheap to keep and expensive to establish. For our gateway, steady state cost about 30 KB of memory and near-zero CPU per socket. A new connection cost a TCP and TLS handshake at the ingress, an HTTP upgrade, token verification, a Redis presence write, a channel subscription, and a database query to backfill missed messages. Measured in staging, one reconnect cost about 400 times the CPU of one minute of idle connection.

Capacity planning for realtime systems usually looks at concurrent connections. The dangerous number is connections per second during a mass disconnect. Anything that closes many sockets at once creates that spike: a pod restart, a node drain, an ingress reload, a load balancer idle timeout aligned across clients, or a mobile network blip.

Synchronised clients are what turn a spike into a storm. If every client waits the same delay, their retries stay aligned. If failed handshakes are retried faster than the backend recovers, the load never drops below capacity.

Environment

Minimal reproduction

The client logic that caused it, cut down:

// old-client.js
function connect() {
  const ws = new WebSocket("wss://rt.example.com/socket");
  ws.onopen = () => ws.send(JSON.stringify({ type: "resume", since: lastId }));
  ws.onclose = () => setTimeout(connect, 1000); // every client, same delay
}
connect();

And the server shutdown:

// old-server.js (excerpt)
process.on("SIGTERM", () => process.exit(0));

Who is affected

Any service holding many long-lived connections per instance: WebSocket gateways, SSE endpoints, MQTT brokers, gRPC streams. The risk grows with connections per instance and with the cost of the reconnect path. Clients with fixed delays or no handshake timeout backoff are the multiplier.