#backups

Databases Bug Report

Our restored MySQL Docker volume would not start: a tar backup of a live InnoDB data directory

Le Minh Khoa · Apr 24, 2026 · 7 min read

When a small self-hosted service lost its disk, we restored the MySQL Docker volume from last night's tar backup, and MySQL refused to start with InnoDB corruption errors. The backup had been a file copy of a running database. How we recovered from an older dump, and the backup approach and restore drills we use now.

Databases Bug Report

Nightly pg_dump looked fine for months until restore day: backups that would not restore

John Nguyen · Apr 9, 2026 · 10 min read

Our nightly pg_dump job reported success every night for four months. On the day we needed one, every dump in the bucket was truncated. A replica query cancellation plus a missing pipefail hid it. Here is the postmortem and the restore-tested backup job we run now.