Our restored MySQL Docker volume would not start: a tar backup of a live InnoDB data directory

A small internal service, a ticketing tool for our support team, ran on a single VM with Docker Compose: the application container and a MySQL container with a named volume. A nightly cron job backed up the volume with a common recipe:

docker run --rm -v ticketing_mysql:/data -v /backups:/backup alpine \
  tar czf /backup/mysql-$(date +%F).tar.gz -C /data .

When the VM's disk failed, we created a new VM, restored the archive into a fresh volume and started the stack. MySQL did not come up:

[ERROR] [MY-012153] [InnoDB] Trying to access page number 4294967295 in space 0, space name innodb_system,
which is outside the tablespace bounds.
[ERROR] [MY-011906] [InnoDB] Database page corruption on disk or a failed file read of page
[page id: space=27, page number=402]. You may have to recover from a backup.
[ERROR] [MY-010119] [Server] Aborting

Was the archive damaged?

gzip -t mysql-2026-04-21.tar.gz && echo "archive ok"
tar tzf mysql-2026-04-21.tar.gz | head

The archive was intact and complete. The files inside were simply not a consistent database.

Older archives

We tried the previous six nightly archives. Two started cleanly, four did not. The ones that worked were from nights when the support team had been idle at backup time.

What we ruled out

• MySQL version mismatch. The same image tag (mysql:8.0.41) was used for backup and restore.
• File ownership. We restored with ownership preserved; permission errors look different.
• Disk errors on the new VM. A fresh MySQL container on the same disk initialised fine.

Root cause

The backup copied the files of a running InnoDB database. InnoDB writes to data files, redo logs and the doublewrite buffer continuously and in a specific order, and relies on its logs to make the files consistent after a crash. A file-level copy taken while the server is writing captures different files, and even different parts of the same file, at different moments. The result is not the state of the database at any point in time, and crash recovery cannot always repair it.

Why "crash consistent" is not enough with tar

InnoDB is designed to survive a crash: after a power loss, the data files plus the redo log let it recover to a consistent state. A crash leaves all files frozen at the same instant. tar reads files one after another over seconds or minutes while the database keeps writing, so the redo log in the archive may be from 02:00:03 and a data file from 02:00:41. That combination never existed, so recovery can fail.

Filesystem or block-level snapshots (LVM, ZFS, cloud volume snapshots) capture all files at one instant and are crash consistent, which is why they work much better than file copies, though they still rely on crash recovery.

Backup methods for MySQL

Who is affected

Anyone backing up database containers by copying their volumes while the database runs: MySQL, MariaDB, PostgreSQL and MongoDB are all affected in the same way. Generic "back up all Docker volumes" scripts are the usual source, because they treat database volumes like any other files.

Environment

Recovery

We restored the newest archive that started cleanly (three days old), then replayed what we could from the email system that feeds tickets. Three days of ticket comments were lost. That was the real cost of a backup that had never been restored.

Logical backups with mysqldump

For a 6 GB database, a logical dump is simple and fast enough:

docker compose exec -T mysql sh -c \
  'exec mysqldump --single-transaction --routines --triggers --events --set-gtid-purged=OFF \
   -uroot -p"$MYSQL_ROOT_PASSWORD" --databases ticketing' \
  | gzip > /backups/ticketing-$(date +%F).sql.gz

--single-transaction takes a consistent snapshot of InnoDB tables without locking them, so the service keeps running during the backup. The dump goes to a file on the host and is then copied off the host to object storage with versioning, because backups on the same disk as the database do not survive a disk failure.

Physical backups for larger databases

For our larger MySQL servers, we use Percona XtraBackup, which copies InnoDB files while the server runs and records the redo log in parallel, then "prepares" the backup into a consistent state:

xtrabackup --backup --target-dir=/backups/full-$(date +%F) --user=backup --password="$BACKUP_PW"
xtrabackup --prepare --target-dir=/backups/full-$(date +%F)

Restore drills

The most important change: a weekly job restores the latest backup into a temporary container and checks it:

docker run -d --name restore-test -e MYSQL_ROOT_PASSWORD=test mysql:8.0.41
until docker exec restore-test mysqladmin -uroot -ptest ping --silent; do sleep 2; done
gunzip -c /backups/ticketing-latest.sql.gz | docker exec -i restore-test mysql -uroot -ptest
docker exec restore-test mysql -uroot -ptest -e \
  "SELECT count(*), max(created_at) FROM ticketing.tickets;"
docker rm -f restore-test

The job alerts if the restore fails or if the newest ticket is older than 26 hours, which catches both broken backups and backups that silently stopped running.

Alternatives and trade-offs

• Stop the container for the backup. Makes a file copy consistent, at the cost of nightly downtime.
• Managed database service with automated backups and point-in-time recovery. For anything important, this removes most of the problem; the ticketing tool moved to one later in the year.

How to test the fix

The restore drill above is the test. We also ran it once against a backup taken during a synthetic write load, to make sure --single-transaction produced a consistent dump while the database was busy.

Confirming in production

The weekly restore drill has passed every week since, with restore time around four minutes. The backup files are copied to versioned object storage in another region, with a lifecycle rule keeping 35 days.

Regression guard

• Generic volume backup scripts exclude database volumes, which have their own backup jobs.
• An alert if no successful backup and no successful restore drill in the last 26 hours and 8 days respectively.

Security implications

Backups contain all customer support conversations, including personal data. Dumps are encrypted before upload, the bucket has object lock enabled so backups cannot be deleted by a compromised host, and the backup database user has only the privileges needed for dumping. Restore drills run in an isolated network and the test container is removed afterwards.

Prevention checklist

• Never copy a running database's files as a backup.
• Use mysqldump --single-transaction, XtraBackup or crash-consistent snapshots.
• Store backups off the host, versioned and protected from deletion.
• Restore backups automatically and check the data, every week.
• Monitor both backup success and restore success.

References

• MySQL 8.0, Backup and recovery: https://dev.mysql.com/doc/refman/8.0/en/backup-and-recovery.html
• MySQL 8.0, mysqldump: https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html
• Percona XtraBackup documentation: https://docs.percona.com/percona-xtrabackup/8.0/
• Docker, Back up, restore, or migrate data volumes: https://docs.docker.com/engine/storage/volumes/#back-up-restore-or-migrate-data-volumes