"No space left on device" with 30% disk free: inodes exhausted by millions of session files

The first alert was from the deploy pipeline: the release step failed while unpacking an archive on one of our web servers.

tar: app/vendor/symfony/console/Command/Command.php: Cannot open: No space left on device
tar: Exiting with failure status due to previous errors

A few minutes later, customer support reported that profile photo uploads were failing. The PHP error log on the same server had:

PHP Warning:  file_put_contents(/var/www/storage/app/tmp/upload_7f3a.bin): Failed to open stream:
No space left on device in /var/www/app/Http/Controllers/AvatarController.php on line 41

The obvious move was to check disk space:

df -h /

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1   80G   55G   25G  70% /

Twenty-five gigabytes free. The kernel was refusing to create files on a disk that was clearly not full.

What we ruled out

• Deleted files still held open. That case shows the opposite pattern, df full while du finds much less, and is covered in post 302. Here df itself said there was space. lsof +L1 showed nothing large anyway.
• A full secondary mount. /var/www was on the root filesystem, not a separate volume.
• Reserved blocks for root. ext4 reserves 5% for root by default, which can make non-root writes fail near 95%. We were at 70%.
• Quotas. No user or project quotas were configured.

The real number

Every file on ext4 needs an inode, a fixed-size structure that stores its metadata. The number of inodes is decided when the filesystem is created. When they run out, you cannot create files, no matter how many free blocks remain.

df -i /

Filesystem       Inodes   IUsed IFree IUse% Mounted on
/dev/nvme0n1p1 10485760 10485760     0  100% /

All 10.4 million inodes were used.

Finding where they went

Counting files per directory needs care on a filesystem with millions of entries, because recursive listings can take a long time and hammer the disk. We used a single find that prints only directory names of each file, then counted:

sudo find / -xdev -type f -printf '%h\n' 2>/dev/null \
  | sort | uniq -c | sort -rn | head -5

9612345 /var/lib/php/sessions
 210331 /var/www/storage/framework/cache/data/4f/2a
  84512 /usr/share/locale
  41022 /var/www/storage/logs
  18877 /usr/lib/python3/dist-packages

On GNU coreutils 8.22 and later, du --inodes gives a quicker overview by directory:

sudo du --inodes -x -d 3 / 2>/dev/null | sort -rn | head

Over 9.6 million files were in /var/lib/php/sessions, each a few hundred bytes.

Root cause

PHP stored sessions as files, and nothing was deleting old ones. On Debian and Ubuntu, PHP's own probabilistic garbage collection is disabled (session.gc_probability = 0) because the distribution ships a cron job and systemd timer that clean the session directory instead. Our server had been migrated from an older image, and the custom PHP-FPM build we deployed did not install that cleanup timer. Every visitor, including every bot, created a session file that was never removed. Tiny files use few blocks but one inode each, so inodes ran out long before space did.

Blocks versus inodes

An ext4 filesystem has two separate resources:

• Data blocks store file contents. df -h reports these.
• Inodes store metadata: owner, permissions, timestamps, pointers to blocks. Each file and directory uses one. df -i reports these.

By default, mkfs.ext4 creates one inode per 16 KiB of disk space (the inode_ratio in /etc/mke2fs.conf). At that ratio an 80 GB filesystem gets about 5.2 million inodes; our image had been built with an 8 KiB ratio, giving 10.4 million. Workloads with millions of files much smaller than the ratio, such as session stores, caches and mail spools, exhaust inodes first.

XFS allocates inodes dynamically up to a percentage of the filesystem, so it rarely hits this limit, but it can still happen when the filesystem is nearly full.

Environment

Who is affected

Any ext4 server that creates many small files without cleanup: file-based sessions, file caches, thumbnail directories, maildir spools, container layers with many small files, and build caches.

Step 1: free inodes now, without freezing the server

Running rm /var/lib/php/sessions/* fails with "Argument list too long" and, even if it worked, would build a huge argument list first. find -delete removes files as it walks the directory. We deleted only sessions older than the configured lifetime, at low I/O priority, so active users stayed logged in:

sudo ionice -c3 nice -n19 find /var/lib/php/sessions -type f -name 'sess_*' -mmin +1440 -delete

It took 23 minutes and brought inode usage down to 9%:

df -i /
# /dev/nvme0n1p1 10485760 958201 9527559 9% /

Deploys and uploads worked again immediately.

Step 2: restore session cleanup

We reinstated a cleanup timer matching the distribution's behaviour, using the configured session.gc_maxlifetime:

# /etc/systemd/system/php-session-clean.service
[Unit]
Description=Clean expired PHP sessions

[Service]
Type=oneshot
Nice=19
IOSchedulingClass=idle
ExecStart=/usr/bin/find /var/lib/php/sessions -type f -name 'sess_*' -mmin +1440 -delete

# /etc/systemd/system/php-session-clean.timer
[Unit]
Description=Run PHP session cleanup every 30 minutes

[Timer]
OnCalendar=*:0/30
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now php-session-clean.timer
systemctl list-timers php-session-clean.timer

Step 3: move sessions off the filesystem

File sessions also break once you run more than one web server. The application already had Redis for cache, so we moved sessions there, where expiry is built in:

; php.ini (or the framework's session config)
session.save_handler = redis
session.save_path = "tcp://10.0.2.15:6379?database=2"
session.gc_maxlifetime = 86400

We also stopped creating sessions for anonymous requests that did not need one, such as health checks and static API calls, which cut session creation by more than half.

Alternatives and trade-offs

• Recreate the filesystem with more inodes (mkfs.ext4 -N or a smaller -i ratio). Valid for storage that will hold many small files by design, but requires a migration.
• XFS for new volumes that hold many small files.
• Enable PHP's own GC (session.gc_probability = 1). Works, but adds latency to random requests and still scans a large directory.

How to test the fix

On a staging server, generate sessions and confirm the timer removes expired ones:

for i in $(seq 1 1000); do curl -s -o /dev/null https://staging.example.com/; done
sudo touch -d '2 days ago' /var/lib/php/sessions/sess_*
sudo systemctl start php-session-clean.service
ls /var/lib/php/sessions | wc -l   # expect 0 (or only fresh sessions)

Confirming in production

A week later, inode use on the web servers stayed between 8% and 10%, and the session directory held fewer than 2,000 files after the move to Redis.

Regression guard

We already alerted on disk space; we added the same for inodes. With node_exporter metrics:

(1 - node_filesystem_files_free{fstype=~"ext4|xfs"} / node_filesystem_files{fstype=~"ext4|xfs"}) > 0.85

The server image build now fails if the session cleanup timer is missing, and the configuration management role asserts session.save_handler = redis.

Security implications

File-based session stores that grow without limit are also a denial of service vector: an attacker can create sessions quickly and exhaust inodes, taking down anything else on the filesystem. Expiry, avoiding sessions for anonymous requests, and rate limiting close that path. Moving sessions to Redis means session data now travels over the network; it runs on a private subnet with authentication enabled.

Prevention checklist

• Monitor inode usage (df -i) alongside disk space on every filesystem.
• Give every directory that accumulates files an owner and a cleanup mechanism.
• Prefer Redis or a database for sessions and caches over local files.
• Choose filesystem inode ratios deliberately for small-file workloads.
• Delete large file sets with find -delete at low I/O priority, not shell globs.

References

• ext4 documentation, inode table and mke2fs: https://www.kernel.org/doc/html/latest/filesystems/ext4/index.html
• mke2fs.conf manual: https://man7.org/linux/man-pages/man5/mke2fs.conf.5.html
• PHP manual, session runtime configuration: https://www.php.net/manual/en/session.configuration.php
• systemd.timer manual: https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html