Service won't restart with EADDRINUSE: an orphaned node process held the port, not TIME_WAIT

A deploy on a Friday afternoon, the kind that usually takes two minutes, left one of our API servers down. The systemd unit kept restarting and failing:

$ journalctl -u orders-api -n 20 --no-pager
Dec 27 16:02:11 api-02 node[48211]: Error: listen EADDRINUSE: address already in use :::3000
Dec 27 16:02:11 api-02 node[48211]:     at Server.setupListenHandle [as _listen2] (node:net:1904:16)
Dec 27 16:02:11 api-02 systemd[1]: orders-api.service: Main process exited, code=exited, status=1/FAILURE
Dec 27 16:02:11 api-02 systemd[1]: orders-api.service: Failed with result 'exit-code'.

The other three servers had restarted fine with the same release.

The first theory: TIME_WAIT

Someone in the channel said what many of us would have said: "It's TIME_WAIT, the old connections are still closing, set SO_REUSEADDR." That theory is common enough to be worth testing properly instead of accepting.

ss -tan state time-wait '( sport = :3000 )' | wc -l

1843

Plenty of TIME_WAIT sockets. But those are the remains of closed connections. The question is whether anything is still listening:

sudo ss -ltnp '( sport = :3000 )'

State  Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0      511                *:3000            *:*     users:(("node",pid=31877,fd=21))

A live listener, owned by PID 31877. The new process could not bind because another process was already bound to the port.

Who is PID 31877?

ps -o pid,ppid,etime,user,cmd -p 31877

  PID  PPID     ELAPSED USER   CMD
31877     1  3-02:41:19 orders node dist/server.js

sudo ls -l /proc/31877/cwd
# lrwxrwxrwx 1 orders orders 0 Dec 27 16:03 /proc/31877/cwd -> /srv/orders-api/releases/2412

A process started three days ago from the previous release directory, with parent PID 1. It was the old server, re-parented to init when the process that started it exited.

What we ruled out

• TIME_WAIT preventing bind. Node.js sets SO_REUSEADDR on listening sockets by default on Linux, and TIME_WAIT connections do not block a new listener with that option. The live listener did.
• Another service configured on port 3000. The command line showed it was our own code.
• A failed deploy step. The new release was complete; the problem was the old one not being gone.

Root cause

The systemd unit started the API with ExecStart=/usr/bin/npm run start, so systemd's main process was npm, and npm spawned node dist/server.js as a child. The unit had been written years earlier with KillMode=process, which tells systemd to signal only the main process on stop. On restart, systemd stopped npm; npm exited without waiting for its child on this server (the other servers happened to forward the signal in time), and the node process, busy with a long report request, carried on. It was re-parented to PID 1, still in the old release directory, still holding the listening socket. The new npm started a new node, which could not bind.

A listener versus TIME_WAIT

TIME_WAIT is a state of a closed TCP connection, kept for a short period so that delayed packets do not get confused with a new connection using the same address pair. On Linux it does not stop a new listening socket from binding to the port when the program sets SO_REUSEADDR, which most servers and runtimes, including Node.js, nginx and Go's standard library on Linux, do by default.

EADDRINUSE on a modern server almost always means one of:

1. another process is listening on the same address and port;
2. the same program started twice (two units, a manual run plus a service);
3. a program without SO_REUSEADDR restarting with connections in TIME_WAIT (rare today);
4. binding to a specific address that is already taken by a wildcard listener.

The first question is always "is something listening?", answered with ss -ltnp.

How systemd stops services

KillMode decides which processes get the stop signal:

With process, children that ignore or outlive the main process's exit stay alive. They usually still sit in the unit's cgroup, which systemctl status shows, but nothing will stop them until they exit by themselves.

Environment

Immediate recovery

sudo kill -TERM 31877
sleep 5
sudo ss -ltnp '( sport = :3000 )'     # empty
sudo systemctl restart orders-api

Fix the unit

We start node directly, so it is the main process systemd tracks, let systemd manage the whole process tree, and give the server time to finish in-flight requests:

 [Service]
 User=orders
 WorkingDirectory=/srv/orders-api/current
-ExecStart=/usr/bin/npm run start
+ExecStart=/usr/bin/node dist/server.js
-KillMode=process
-TimeoutStopSec=5
+KillMode=mixed
+KillSignal=SIGTERM
+TimeoutStopSec=30
 Restart=on-failure

With mixed, the main process gets SIGTERM and can drain; after 30 seconds, anything still alive in the cgroup gets SIGKILL. No orphan survives a stop, even if some future change adds child processes.

Graceful shutdown in the application

The server now handles SIGTERM by closing the listener first, then exiting once in-flight requests finish:

// server.js
const server = app.listen(3000);

process.on('SIGTERM', () => {
  server.close(() => process.exit(0));              // stop accepting, finish in-flight requests
  server.closeIdleConnections();                     // drop idle keep-alive sockets (Node 18.2+)
  setTimeout(() => process.exit(1), 25_000).unref(); // hard stop before systemd's SIGKILL
});

Alternatives and trade-offs

• SO_REUSEPORT lets several processes bind the same port. It is useful for intentional zero-downtime handovers, but here it would have hidden the bug and split traffic between old and new code.
• systemd socket activation passes a listening socket owned by systemd to the service, so restarts never race for the port. A good fit for single-process servers.
• Keeping npm as a wrapper. Process managers and package scripts in ExecStart hide the real server from systemd; avoid them in units.
• fuser -k 3000/tcp in ExecStartPre. A blunt workaround that kills whatever holds the port. It treats the symptom and can kill something unrelated.

How to test the fix

On staging, start a long request, restart the service and confirm nothing from the old release survives:

curl -s https://staging.example.com/api/slow-report?seconds=20 &
sudo systemctl restart orders-api
systemd-cgls -u orders-api.service          # only new processes listed
for p in $(pgrep -u orders node); do readlink /proc/$p/cwd; done   # only the current release

Confirming in production

Over the next month of deploys, every restart completed, and a deploy hook now checks that the listener on port 3000 belongs to the new main PID:

main=$(systemctl show -p MainPID --value orders-api)
owner=$(ss -ltnpH '( sport = :3000 )' | grep -oP 'pid=\K[0-9]+' | head -1)
[ "$owner" = "$main" ] || { echo "port 3000 owned by pid $owner, expected $main"; exit 1; }

Regression guard

• A lint step in our configuration repository rejects KillMode=process, KillMode=none and package-manager wrappers in ExecStart.
• The deploy hook above fails the deploy if a stale process owns the port.

Security implications

The orphaned process kept serving requests with the previous release's code, which included a dependency we had just upgraded for a security fix. Stale processes after deploys can silently keep vulnerable code running. Killing the whole cgroup on stop is therefore a security property as well as an availability one.

Prevention checklist

• On EADDRINUSE, check for a live listener with ss -ltnp before blaming TIME_WAIT.
• Start the real server binary in ExecStart and keep KillMode=control-group or mixed.
• Implement graceful shutdown on SIGTERM with a timeout.
• Verify after deploys that the port owner is the new process.
• Consider socket activation for services that must restart without races.

References

• systemd.kill manual, KillMode: https://www.freedesktop.org/software/systemd/man/latest/systemd.kill.html
• ss manual: https://man7.org/linux/man-pages/man8/ss.8.html
• socket(7), SO_REUSEADDR and SO_REUSEPORT: https://man7.org/linux/man-pages/man7/socket.7.html
• Node.js http server.close and closeIdleConnections: https://nodejs.org/docs/latest-v20.x/api/http.html