📑 Daftar Isi
- Introduction
- What Is an Inode?
- Symptoms of Inode Exhaustion
- How to Check Inode Usage
- Command #1: Check Server Inode Usage
- Command #2: Find the Largest Error Logs
- Breakdown of Each Part
- Example Output
- Command #3: Find the Directory Consuming the Most Inodes
- Example Output
- Command #4: Check Inodes by File Type
- Example Output
- How to Fix Inode Exhaustion
- ⚠️ SECURITY WARNING: Backup Before Cleanup
- Solution 1: Delete Unneeded Error Logs
- Solution 2: Clean WordPress Cache
- Solution 3: Clean Temporary and Session Files
- Solution 4: Clean Old Backups and Archives
- Solution 5: Clean cPanel and WHM Cache
- Solution 6: Configure Logrotate
- Pro Tips: Preventing Inode Exhaustion
- FAQ
- How do I increase inode quota on a cPanel server?
- Is it safe to delete error_log?
- Why does df -h show space is still free but I can't create new files?
- How many inodes are safe for a cPanel server?
- Related Issues
- Conclusion
Introduction
A while back I got an emergency call from a client: “All websites are dead! cPanel is inaccessible, even SSH is timing out!” When I checked from the server side, the situation was worse than I imagined — inode usage was at 100%. Not disk space that was full, but the number of inodes that had been exhausted. And when inodes run out, you can’t create new files — including log files, session files, temporary files, even configuration files needed by the web server and other services.
What made the situation more critical was that this client hosted 40+ websites on a single shared hosting account. One unmaintained website — with an error_log that kept growing without rotation — was enough to consume the entire server’s inode quota. In less than 6 hours, every website on that server was completely down.
From that experience, I learned one thing: inode exhaustion is more dangerous than running out of disk space. Because when disk space runs out, you can still delete old files to make room. But when inodes run out, you can’t even delete files — because the deletion process itself requires inodes. The only way is to find which directory is consuming the most inodes, then aggressively reduce it.
What Is an Inode?
Before troubleshooting, it’s important to understand what an inodes is and why inodes can run out.
Inodes Are the “Identity” of Every File
In Linux/Unix, every file and directory has a unique number called an inode. The inode stores information about the file: who owns it, when it was created, when it was last accessed, its size, and where its data blocks are located on disk. The file name you see in ls is actually just a “label” pointing to a specific inode.
The analogy is simple: imagine an apartment building. Each apartment unit has a number (that’s the inode). The owner’s name (file name) can change, but the unit number (inode) stays the same. When the building runs out of unit numbers, even if there’s empty space in the hallway, you can’t add new units.
Inodes vs. Disk Space
Many people mistakenly think “disk space full” and “inodes exhausted” are the same thing. They’re not. They’re two different problems:
- Disk space full — the total size of all files has reached the disk capacity limit. Large files like videos, backups, or databases are what consume space.
- Inodes exhausted — the number of files has reached the limit, even though the total file size is still small. Thousands of small files (logs, cache, sessions, temporary files) can exhaust inodes without consuming much space.
A concrete example: a single 1GB log file uses only 1 inode. But 1 million 1KB log files each use 1 million inodes. This is why cPanel servers hosting many websites are very vulnerable to inode exhaustion — each website can generate hundreds of small files per day (logs, cache, sessions).
Inode Quota on cPanel Hosting
On cPanel hosting, each account is typically given a specific inode quota — for example 250,000 or 500,000 inodes. This quota is set by the hosting provider and can be checked from cPanel → Quota Usage. When the number of inodes reaches the quota limit, you can’t create new files until you reduce the existing file count.
Symptoms of Inode Exhaustion
Inode exhaustion has very distinctive symptoms that are easy to recognize if you know what to look for:
- “No space left on device” error even though disk space still exists — the most classic symptom.
df -hshows disk space is still 50% free, but you can’t create new files. - cPanel/WHM inaccessible — cPanel needs temporary files to operate. When inodes run out, cPanel can’t create temporary files and eventually crashes.
- Web server (Apache/Nginx) dies — the web server needs log files and temporary files. Without inodes, the web server can’t write logs and eventually stops.
- MySQL/MariaDB dies — the database server needs temporary files for queries and sorting. When inodes run out, database operations fail.
- Email can’t be sent/received — the mail server needs queue files and logs. Without inodes, emails get stuck in the queue and are eventually discarded.
- SSH timeout or can’t login — some systems need temporary files for SSH authentication. When inodes run out, SSH is also affected.
- “Too many links” error on directories — directories with a very large number of files can hit link limits, which is also related to inodes.
How to Check Inode Usage
Command #1: Check Server Inode Usage
The first and most important command is checking how many inodes are used and how many are still available:
df -i
Example output when normal:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 655360 452340 203020 69% /
tmpfs 40960 2 40958 1% /dev/shm
Example output when inodes are exhausted:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 655360 655360 0 100% /
tmpfs 40960 2 40958 1% /dev/shm
Pay attention to the IFree and IUse% columns. When IFree = 0 and IUse% = 100%, your server has run out of inodes. All file creation operations will fail.
Command #2: Find the Largest Error Logs
One of the most common causes of inode exhaustion on cPanel servers is unrotated error_log files. Error_log files that keep growing without limits can consume thousands of inodes. Here’s the command to find the largest error_log files:
find . -name 'error_log' -exec du -ch {} + | grep total
Let’s break down this command:
Breakdown of Each Part
-
find .— Start searching from the current directory (.). You can replace this with a specific path like/home/usernameor/homefor a broader search. -
-name 'error_log'— Filter the search: only files named exactlyerror_log. This catches error log files from PHP, Apache, and other applications using this default name. -
-exec du -ch {} +— For each found file, rundu -ch(disk usage, cumulative, human-readable). The-cflag adds a total line at the end,-hmakes output in an easy-to-read format (KB, MB, GB). -
| grep total— Only show lines containing “total” — the summary line showing the total size of all error_log files found.
Example Output
find . -name 'error_log' -exec du -ch {} + | grep total
4.2G ./public_html/error_log
1.8G ./public_html/subdomain/error_log
892M ./public_html/blog/error_log
127M ./public_html/forum/error_log
7.0G total
From this output, it’s crystal clear: there’s 7GB total of error_log files on this server. These files are what’s consuming the inodes. Note that each error_log also adds to the inode count — each file occupies 1 inode, and in the same directory, each subdirectory also occupies 1 inode.
Command #3: Find the Directory Consuming the Most Inodes
Besides error_log, there are many other types of files that can consume inodes. To find which directory is holding the most inodes, use this command:
find . -xdev -printf '%hn' | sort | uniq -c | sort -rn | head -10
Explanation:
find . -xdev— Search for all files in the current directory, but don’t cross into other filesystems (-xdev). This prevents searching into mounted directories like/procor/dev.-printf '%hn'— For each found file, print its parent directory name (%h= head/path). So we get a list of directories, one line per file.| sort | uniq -c— Count occurrences of each directory — i.e., the number of files in that directory.| sort -rn | head -10— Sort from highest to lowest and show the top 10.
Example Output
127845 /home/username/public_html/wp-content/cache
89234 /home/username/public_html/wp-content/uploads
45678 /home/username/public_html/tmp
23456 /home/username/public_html/wp-content/plugins/akismet
12345 /home/username/public_html/error_logs
8901 /home/username/public_html/wp-content/uploads/2024
5678 /home/username/public_html/wp-content/uploads/2023
3456 /home/username/public_html/logs
2345 /home/username/public_html/tmp/sessions
1234 /home/username/public_html/cache
From this output, the wp-content/cache directory is holding 127,845 inodes — that’s nearly half of the 250,000 inode quota! This is a WordPress caching directory that was never cleaned up.
Command #4: Check Inodes by File Type
To understand which file types are consuming the most inodes:
find . -xdev -type f | sed 's/.*.//' | sort | uniq -c | sort -rn | head -10
Explanation:
find . -xdev -type f— Find all files (not directories) on the same filesystem.sed 's/.*.//'— Extract the file extension from the path. For example,/path/to/error.logbecomeslog.sort | uniq -c | sort -rn | head -10— Count and sort by highest number.
Example Output
234567 log
123456 php
89012 txt
56789 html
34567 png
23456 jpg
12345 css
8901 js
5678 xml
3456 json
Files with the .log extension top the list with 234,567 files — already exceeding the 250,000 inode quota. These log files are the primary cause of inode exhaustion.
How to Fix Inode Exhaustion
⚠️ SECURITY WARNING: Backup Before Cleanup
Before performing a mass cleanup, you MUST backup files that are still needed. An incorrect inode cleanup could delete important data that can’t be recovered.
# Backup database
mysqldump --all-databases > /tmp/all-databases-backup-$(date +%Y%m%d-%H%M).sql
# Backup important configuration files
cp -r /home/username/public_html/wp-config.php /tmp/
cp -r /home/username/public_html/.htaccess /tmp/
# Backup file list for reference
find /home/username -type f > /tmp/all-files-list-$(date +%Y%m%d-%H%M).txt
Verify backup succeeded:
ls -lh /tmp/all-databases-backup-*.sql
wc -l /tmp/all-files-list-*.txt
Solution 1: Delete Unneeded Error Logs
This is the quickest and most effective solution for inode exhaustion. Large, unrotated error_log files are the number one cause:
# First check which error_log files are largest
find /home/username -name 'error_log' -exec du -h {} + | sort -rh | head -10
# Delete error_log files that are no longer needed
# ⚠️ MAKE SURE these aren't logs you still need for debugging
find /home/username -name 'error_log' -delete
⚠️ IMPORTANT: Don’t delete error_log if you’re currently debugging a specific issue. Only delete error_log when you’ve finished debugging or the log is truly no longer needed.
Solution 2: Clean WordPress Cache
If you’re using WordPress, the wp-content/cache directory can hold hundreds of thousands of stale cache files:
# Delete all WordPress cache files
find /home/username/public_html/wp-content/cache -type f -delete
# Or delete everything in the cache directory
rm -rf /home/username/public_html/wp-content/cache/*
Verify:
# Check if inodes have decreased
df -i / | tail -1
# Check if the cache directory still exists
ls -la /home/username/public_html/wp-content/cache/
Solution 3: Clean Temporary and Session Files
PHP session files and temporary files can also hold many inodes:
# Delete session files older than 7 days
find /tmp -name "sess_*" -type f -mtime +7 -delete
# Delete old PHP temporary files
find /tmp -name "*.tmp" -type f -mtime +7 -delete
# Delete inactive cache files
find /home/username/public_html/tmp -type f -mtime +30 -delete
Solution 4: Clean Old Backups and Archives
Unneeded backup files also hold inodes:
# Find large backup files
find /home/username -name "*.bak" -o -name "*.backup" -o -name "*.tar.gz" -o -name "*.zip" | head -20
# Delete old backups (older than 30 days)
find /home/username -name "*.bak" -mtime +30 -delete
find /home/username -name "*.backup" -mtime +30 -delete
find /home/username -name "*.tar.gz" -mtime +30 -delete
Solution 5: Clean cPanel and WHM Cache
cPanel and WHM also store cache files that can hold inodes:
# Clean cPanel cache
rm -rf /home/username/.cpanel/cache/*
# Clean ea4 cache (if exists)
rm -rf /home/username/.cpanel/ea4/cache/*
# Clean old cPanel logs
find /home/username/logs -name "*.log" -mtime +30 -delete
Solution 6: Configure Logrotate
After successfully fixing inode exhaustion, the next step is preventing it from happening again. Configure logrotate for all log files:
# Create logrotate configuration for error_log
cat > /etc/logrotate.d/error_log << 'EOF'
/home/*/public_html/error_log {
daily
missingok
rotate 7
compress
delaycompress
notifempty
create 644 nobody nobody
}
EOF
Or if you're using cPanel, you can use cPanel's built-in log rotation feature → cPanel → Metrics → Metrics → Errors.
Pro Tips: Preventing Inode Exhaustion
- Monitor inode usage regularly — set up a cron job for alerts:
0 */6 * * * USAGE=$(df -i / | tail -1 | awk '{print $5}' | tr -d '%'); [ $USAGE -gt 80 ] && echo "ALERT: Inode usage at $USAGE%" | mail -s "Inode Alert" admin@domain.com - Use logrotate for all log files — make sure all actively written logs are configured with logrotate. Especially error_log, access_log, and application log files.
- Clean cache regularly — set up a cron job to delete old cache files:
0 2 * * * find /home/*/public_html/wp-content/cache -type f -mtime +7 -delete - Limit files per directory — some applications like email or file storage can generate thousands of small files. Consider limiting the number of files or using alternative storage.
- Use tmpfs for temporary files — mount
/tmpas tmpfs (RAM-based) for temporary files. Files on tmpfs don't consume inodes on the main filesystem. - Set inode quota per account — if you manage a cPanel server, consider setting per-account inode quotas to prevent one account from consuming all server inodes.
- Audit files regularly — at least once a month, check which directories are holding the most files and clean up what's not needed.
FAQ
How do I increase inode quota on a cPanel server?
Inode quota is usually set by the hosting provider and can't be changed by the user. If you manage your own server (VPS/dedicated), you can increase inode quota by creating a new filesystem and mounting it to the directory that needs more inodes. But the better approach is to reduce unnecessary files — because adding inodes means adding a new filesystem that also requires additional management.
Is it safe to delete error_log?
Yes, deleting error_log is generally safe. Error_log only contains debugging information and errors that have already occurred. After errors are fixed, old error_log files are no longer needed. But make sure you're not currently in the process of debugging a specific issue — because error_log might contain clues needed to find the root cause.
Why does df -h show space is still free but I can't create new files?
Because the problem isn't disk space, but inodes. df -h shows disk space usage (size in bytes). To check inodes, use df -i. If df -i shows IUse% = 100%, then inodes are exhausted even though space is still available.
How many inodes are safe for a cPanel server?
As a rule of thumb, keep inode usage below 80%. If it's approaching 90%, do an audit and cleanup immediately. For shared hosting servers with many accounts, consider setting per-account inode quotas (e.g., 250,000 per account) to prevent one account from consuming all server inodes.
Related Issues
- How to Detect & Free Disk Space from Deleted Files on Linux — deleted files still held by processes can also hold space and inodes
- How to Troubleshoot High Database Load in MariaDB — high database load can trigger many errors that produce large error_log files
- How to Fix Apache Slots Being Full (503 Service Unavailable) — Apache slots filling up often happens alongside inode exhaustion due to constantly growing logs
Conclusion
Inode exhaustion is a problem that's often ignored until it happens — and when it does, the impact can be severe: all websites down, services dead, and you can't even delete files for recovery. With df -i and find . -name 'error_log' -exec du -ch {} + | grep total, you can detect this problem before it's too late.
Most importantly, prevention is always better than treatment. Configure logrotate, monitor inode usage regularly, and clean cache on a regular basis. Don't wait until inodes run out and all websites go down — trust me, I've been through it at 3 AM and don't want to go through it again.