Our nightly pg_dump job reported success every night for four months. On the day we needed one, every dump in the bucket was truncated. A replica query cancellation plus a missing pipefail hid it. Here is the postmortem and the restore-tested backup job we run now.
Our nightly report export ran perfectly when we typed it, and failed every night as a systemd service with node: command not found. Services get no login shell, no .bashrc and a working directory of /. Here is how we made the unit self-contained.
A five-minute cron export started taking six minutes as data grew. Two runs then wrote the same CSV at once, and a partner rejected a file with mangled rows. We serialised the job with flock, bounded it with timeout, and published the file with an atomic rename.
A worker image built fine but every container exited at once with "no such file or directory" for a file that clearly existed. A Windows checkout had turned the shebang into /bin/bash with a carriage return. We fixed it with .gitattributes and a CI check.
Our deploy script printed a pg_dump authentication error, then happily ran migrations and switched traffic to a half-migrated release. Bash does not stop on errors by default, and a pipe hid the failure even where we checked. Here is the strict-mode setup and its traps.