Our nightly report export ran perfectly when we typed it, and failed every night as a systemd service with node: command not found. Services get no login shell, no .bashrc and a working directory of /. Here is how we made the unit self-contained.
A five-minute cron export started taking six minutes as data grew. Two runs then wrote the same CSV at once, and a partner rejected a file with mangled rows. We serialised the job with flock, bounded it with timeout, and published the file with an atomic rename.
A queue worker with Restart=always was found dead in the morning with result start-limit-hit. systemd had stopped restarting it after five crashes in ten seconds while Redis was briefly down. How to read the failure, tune StartLimit and RestartSec, and stop flapping services from staying dead.
Our thumbnail service hit Errno 24 at about 1,000 concurrent connections even though ulimit -n showed 65535. systemd services never read limits.conf. We set LimitNOFILE in a drop-in and fixed a client leak that was eating descriptors.
After a routine deploy our Node.js API failed to start with EADDRINUSE on port 3000. Everyone blamed TIME_WAIT. The real cause was the old node process surviving the restart, because systemd only killed the npm wrapper above it. How we proved it with ss and fixed the unit.