Databases
Bug Report
Nightly pg_dump looked fine for months until restore day: backups that would not restore
Our nightly pg_dump job reported success every night for four months. On the day we needed one, every dump in the bucket was truncated. A replica query cancellation plus a missing pipefail hid it. Here is the postmortem and the restore-tested backup job we run now.