#disk-space

Operating Systems Bug Report

"No space left on device" with 30% disk free: inodes exhausted by millions of session files

Kenji Watanabe · Oct 18, 2025 · 7 min read

Uploads and deploys failed with No space left on device while df showed 30% free. The filesystem had run out of inodes: a session directory held 9.6 million tiny files. How we found it with df -i, cleaned it without locking the box, and stopped it coming back.

Databases Bug Report

MySQL stopped when binary logs filled the disk: retention, purging and alerts

Daniel Tran · Sep 23, 2025 · 9 min read

Our MySQL primary hung and then aborted with the data volume at 100%. Binary logs had grown to 410 GB because a backfill tripled write volume while retention had been raised for a lagging replica. How we recovered, purged safely and stopped it recurring.

Cloud & Infrastructure Bug Report

Linux "no space left on device" from Docker overlay2, logs and build cache

Daniel Tran · Jan 15, 2025 · 8 min read

Our single-host production server stopped accepting writes at 03:12 with no space left on device. Docker had quietly filled the disk with a 31 GB container log, old images and build cache. Here is how we found each piece and stopped it from coming back.