Ahosting Logo
Knowledge Base

Understanding Inodes and Running Out of Them

Two separate quantities, and only one is on the usage screenBytes· what everybody watches· shown everywhere· and often plenty free while writes failInodes· one per file, however small· exhausted by many tiny files: cache, sessions, mail· and every error says the disk is fullWhere they come fromSession files nobody prunes, a cache directory with millions of entries, and mail with one fileper message.

A server refuses to write. Uploads fail, the database errors, mail is rejected. Every message says there is no space left. You check the disk and it is half empty.

The count that ran out is not bytes.

What an inode is

A file system tracks two things independently: how much data it holds, and how many objects it holds.

Every file, directory and link consumes exactly one inode, whatever its size. A one-byte file and a one-gigabyte file each use one.

The inode count is fixed when the file system is created. When it is exhausted, nothing new can be created even with abundant space, and the error the kernel returns is the same one used for a full disk, which is why this sends people in the wrong direction.

The check

df -h
df -i

The first reports bytes. The second reports inodes. If df -i shows 100% on the relevant file system, that is the answer, and nothing about deleting large files will help.

This one command is the whole diagnosis, and it is worth running early whenever a write fails, because it costs a second and rules out an entire category.

Finding what consumed them

You are looking for many small files, not large ones.

for d in /*; do echo -n "$d "; find "$d" -xdev 2>/dev/null | wc -l; done

Then narrow down within whichever directory dominates:

find /var -xdev -type d -exec sh -c 'echo "$(ls -A "$1" | wc -l) $1"' _ {} \; 2>/dev/null | sort -rn | head -20

-xdev keeps the search on one file system, which matters, without it the count wanders into mounted volumes and the result is meaningless.

These commands are slow on a machine already in trouble. Start them and let them run.

The usual culprits

PHP session files. One per visitor, in a single directory, cleaned by a scheduled task that may not be running. On a busy site this reaches millions.

Mail queues. A compromised account sending continuously produces one file per message, and the queue grows faster than anything else on the machine.

Cache directories. Page caches and thumbnail caches generate very large numbers of very small files, which is exactly the pattern that exhausts inodes without using space.

A script writing one file per event. Logging, queueing, temporary output. Anything that creates and never removes.

Clearing safely

Deleting millions of files in one command frequently fails with an argument list error. Use find with a delete action instead of a shell glob:

find /var/lib/php/sessions -type f -mtime +2 -delete

Delete by age rather than everything. Session files still in use belong to logged-in visitors, and removing them logs everyone out at once, which is a support incident on top of the one you are already handling.

Expect it to take a long time. Removing a million files is millions of operations, and the machine is already under pressure.

Preventing the recurrence

Clearing the files fixes today. Whatever created them is still doing it.

Find and repair the mechanism: the session cleanup that is not running, the cache with no expiry, the script with no removal step. Scheduling jobs with cron and systemd timers explains making the cleanup reliable.

And monitor the count rather than only the space. Most monitoring watches bytes and would not have warned you about this at all. Managing VPS resources and monitoring performance deals with adding it.

On shared hosting

Accounts have an inode allowance as well as a disk quota, and the same symptoms appear when it is reached: mail rejected, uploads failing, sessions breaking, with disk usage looking fine.

cPanel shows the file count in its statistics. If several unrelated things fail at once on an account with space to spare, check it before investigating any of them individually. Monitoring your hosting resources picks it up from there.

Counting without waiting for find

Counting millions of files takes a long time on a machine already struggling. Two faster approaches.

df -i # totals per filesystem, instant
du --inodes -d 1 ~ 2>/dev/null | sort -n | tail

The second reports inode usage per directory directly, one level at a time, which narrows the search far faster than walking the whole tree. Descend into whichever directory dominates and repeat.

Where it is not available, restrict find by depth rather than letting it run everywhere:

find ~ -xdev -maxdepth 3 -type d -exec sh -c 'echo "$(ls -A "$1" 2>/dev/null | wc -l) $1"' _ {} \; | sort -rn | head

Deleting millions of files without making it worse

A shell glob fails outright past a certain count, and a plain recursive delete on a directory with millions of entries can occupy the machine for a very long time.

find /path/to/dir -type f -delete
find /path/to/dir -type f -mtime +7 -delete

find -delete handles any number, and adding an age restriction is what makes the operation safe on a live system: deleting only what is genuinely stale rather than everything.

Where an entire directory is disposable, moving it aside and deleting in the background is gentler than deleting in place, and it returns the directory to service immediately:

mv /path/cache /path/cache.old && mkdir /path/cache
nohup rm -rf /path/cache.old >/dev/null 2>&1 &

Filesystems differ

The fixed inode count is a property of the older ext filesystems, where it is decided at creation and cannot be changed afterwards, which is why the only real fix is recreating the filesystem.

XFS allocates inodes dynamically, so it does not run out in the same way. A machine on XFS reporting a space problem genuinely has a space problem.

df -T

Worth checking before spending time on this at all: if the filesystem is XFS, the inode explanation does not apply and the cause is elsewhere.

Watch the count, not just the space

Almost all monitoring reports disk space and not file count, which is why this fault arrives without warning.

df -i | awk 'NR>1 && $5+0 > 80 {print "inode warning: " $6 " at " $5}'

Run daily with the output going somewhere a person reads. It costs nothing and it converts a sudden total failure into a notice a week ahead. Managing VPS resources and monitoring performance goes into adding it alongside the other checks.