• Indonesian
  • English
  • What Is Inode & Fix Inode Exhaustion on cPanel

    Kecepatan:

    Introduction

    A while back I got an emergency call from a client: “All websites are dead! cPanel is inaccessible, even SSH is timing out!” When I checked from the server side, the situation was worse than I imagined — inode usage was at 100%. Not disk space that was full, but the number of inodes that had been exhausted. And when inodes run out, you can’t create new files — including log files, session files, temporary files, even configuration files needed by the web server and other services.

    What made the situation more critical was that this client hosted 40+ websites on a single shared hosting account. One unmaintained website — with an error_log that kept growing without rotation — was enough to consume the entire server’s inode quota. In less than 6 hours, every website on that server was completely down.

    From that experience, I learned one thing: inode exhaustion is more dangerous than running out of disk space. Because when disk space runs out, you can still delete old files to make room. But when inodes run out, you can’t even delete files — because the deletion process itself requires inodes. The only way is to find which directory is consuming the most inodes, then aggressively reduce it.

    What Is an Inode?

    Before troubleshooting, it’s important to understand what an inodes is and why inodes can run out.

    Inodes Are the “Identity” of Every File

    In Linux/Unix, every file and directory has a unique number called an inode. The inode stores information about the file: who owns it, when it was created, when it was last accessed, its size, and where its data blocks are located on disk. The file name you see in ls is actually just a “label” pointing to a specific inode.

    The analogy is simple: imagine an apartment building. Each apartment unit has a number (that’s the inode). The owner’s name (file name) can change, but the unit number (inode) stays the same. When the building runs out of unit numbers, even if there’s empty space in the hallway, you can’t add new units.

    Inodes vs. Disk Space

    Many people mistakenly think “disk space full” and “inodes exhausted” are the same thing. They’re not. They’re two different problems:

    • Disk space full — the total size of all files has reached the disk capacity limit. Large files like videos, backups, or databases are what consume space.
    • Inodes exhausted — the number of files has reached the limit, even though the total file size is still small. Thousands of small files (logs, cache, sessions, temporary files) can exhaust inodes without consuming much space.

    A concrete example: a single 1GB log file uses only 1 inode. But 1 million 1KB log files each use 1 million inodes. This is why cPanel servers hosting many websites are very vulnerable to inode exhaustion — each website can generate hundreds of small files per day (logs, cache, sessions).

    Inode Quota on cPanel Hosting

    On cPanel hosting, each account is typically given a specific inode quota — for example 250,000 or 500,000 inodes. This quota is set by the hosting provider and can be checked from cPanel → Quota Usage. When the number of inodes reaches the quota limit, you can’t create new files until you reduce the existing file count.

    Symptoms of Inode Exhaustion

    Inode exhaustion has very distinctive symptoms that are easy to recognize if you know what to look for:

    • “No space left on device” error even though disk space still exists — the most classic symptom. df -h shows disk space is still 50% free, but you can’t create new files.
    • cPanel/WHM inaccessible — cPanel needs temporary files to operate. When inodes run out, cPanel can’t create temporary files and eventually crashes.
    • Web server (Apache/Nginx) dies — the web server needs log files and temporary files. Without inodes, the web server can’t write logs and eventually stops.
    • MySQL/MariaDB dies — the database server needs temporary files for queries and sorting. When inodes run out, database operations fail.
    • Email can’t be sent/received — the mail server needs queue files and logs. Without inodes, emails get stuck in the queue and are eventually discarded.
    • SSH timeout or can’t login — some systems need temporary files for SSH authentication. When inodes run out, SSH is also affected.
    • “Too many links” error on directories — directories with a very large number of files can hit link limits, which is also related to inodes.

    How to Check Inode Usage

    Command #1: Check Server Inode Usage

    The first and most important command is checking how many inodes are used and how many are still available:

    df -i

    Example output when normal:

    Filesystem      Inodes  IUsed   IFree IUse% Mounted on
    /dev/sda1       655360  452340  203020   69% /
    tmpfs            40960       2   40958    1% /dev/shm

    Example output when inodes are exhausted:

    Filesystem      Inodes  IUsed   IFree IUse% Mounted on
    /dev/sda1       655360  655360      0  100% /
    tmpfs            40960       2   40958    1% /dev/shm

    Pay attention to the IFree and IUse% columns. When IFree = 0 and IUse% = 100%, your server has run out of inodes. All file creation operations will fail.

    Command #2: Find the Largest Error Logs

    One of the most common causes of inode exhaustion on cPanel servers is unrotated error_log files. Error_log files that keep growing without limits can consume thousands of inodes. Here’s the command to find the largest error_log files:

    find . -name 'error_log' -exec du -ch {} + | grep total

    Let’s break down this command:

    Breakdown of Each Part

    1. find . — Start searching from the current directory (.). You can replace this with a specific path like /home/username or /home for a broader search.

    2. -name 'error_log' — Filter the search: only files named exactly error_log. This catches error log files from PHP, Apache, and other applications using this default name.

    3. -exec du -ch {} + — For each found file, run du -ch (disk usage, cumulative, human-readable). The -c flag adds a total line at the end, -h makes output in an easy-to-read format (KB, MB, GB).

    4. | grep total — Only show lines containing “total” — the summary line showing the total size of all error_log files found.

    Example Output

    find . -name 'error_log' -exec du -ch {} + | grep total
     4.2G    ./public_html/error_log
     1.8G    ./public_html/subdomain/error_log
     892M    ./public_html/blog/error_log
     127M    ./public_html/forum/error_log
     7.0G    total

    From this output, it’s crystal clear: there’s 7GB total of error_log files on this server. These files are what’s consuming the inodes. Note that each error_log also adds to the inode count — each file occupies 1 inode, and in the same directory, each subdirectory also occupies 1 inode.

    Command #3: Find the Directory Consuming the Most Inodes

    Besides error_log, there are many other types of files that can consume inodes. To find which directory is holding the most inodes, use this command:

    find . -xdev -printf '%hn' | sort | uniq -c | sort -rn | head -10

    Explanation:

    • find . -xdev — Search for all files in the current directory, but don’t cross into other filesystems (-xdev). This prevents searching into mounted directories like /proc or /dev.
    • -printf '%hn' — For each found file, print its parent directory name (%h = head/path). So we get a list of directories, one line per file.
    • | sort | uniq -c — Count occurrences of each directory — i.e., the number of files in that directory.
    • | sort -rn | head -10 — Sort from highest to lowest and show the top 10.

    Example Output

      127845 /home/username/public_html/wp-content/cache
       89234 /home/username/public_html/wp-content/uploads
       45678 /home/username/public_html/tmp
       23456 /home/username/public_html/wp-content/plugins/akismet
       12345 /home/username/public_html/error_logs
        8901 /home/username/public_html/wp-content/uploads/2024
        5678 /home/username/public_html/wp-content/uploads/2023
        3456 /home/username/public_html/logs
        2345 /home/username/public_html/tmp/sessions
        1234 /home/username/public_html/cache

    From this output, the wp-content/cache directory is holding 127,845 inodes — that’s nearly half of the 250,000 inode quota! This is a WordPress caching directory that was never cleaned up.

    Command #4: Check Inodes by File Type

    To understand which file types are consuming the most inodes:

    find . -xdev -type f | sed 's/.*.//' | sort | uniq -c | sort -rn | head -10

    Explanation:

    • find . -xdev -type f — Find all files (not directories) on the same filesystem.
    • sed 's/.*.//' — Extract the file extension from the path. For example, /path/to/error.log becomes log.
    • sort | uniq -c | sort -rn | head -10 — Count and sort by highest number.

    Example Output

      234567 log
      123456 php
       89012 txt
       56789 html
       34567 png
       23456 jpg
       12345 css
        8901 js
        5678 xml
        3456 json

    Files with the .log extension top the list with 234,567 files — already exceeding the 250,000 inode quota. These log files are the primary cause of inode exhaustion.

    How to Fix Inode Exhaustion

    ⚠️ SECURITY WARNING: Backup Before Cleanup

    Before performing a mass cleanup, you MUST backup files that are still needed. An incorrect inode cleanup could delete important data that can’t be recovered.

    # Backup database
    mysqldump --all-databases > /tmp/all-databases-backup-$(date +%Y%m%d-%H%M).sql
    
    # Backup important configuration files
    cp -r /home/username/public_html/wp-config.php /tmp/
    cp -r /home/username/public_html/.htaccess /tmp/
    
    # Backup file list for reference
    find /home/username -type f > /tmp/all-files-list-$(date +%Y%m%d-%H%M).txt

    Verify backup succeeded:

    ls -lh /tmp/all-databases-backup-*.sql
    wc -l /tmp/all-files-list-*.txt

    Solution 1: Delete Unneeded Error Logs

    This is the quickest and most effective solution for inode exhaustion. Large, unrotated error_log files are the number one cause:

    # First check which error_log files are largest
    find /home/username -name 'error_log' -exec du -h {} + | sort -rh | head -10
    
    # Delete error_log files that are no longer needed
    # ⚠️ MAKE SURE these aren't logs you still need for debugging
    find /home/username -name 'error_log' -delete

    ⚠️ IMPORTANT: Don’t delete error_log if you’re currently debugging a specific issue. Only delete error_log when you’ve finished debugging or the log is truly no longer needed.

    Solution 2: Clean WordPress Cache

    If you’re using WordPress, the wp-content/cache directory can hold hundreds of thousands of stale cache files:

    # Delete all WordPress cache files
    find /home/username/public_html/wp-content/cache -type f -delete
    
    # Or delete everything in the cache directory
    rm -rf /home/username/public_html/wp-content/cache/*

    Verify:

    # Check if inodes have decreased
    df -i / | tail -1
    
    # Check if the cache directory still exists
    ls -la /home/username/public_html/wp-content/cache/

    Solution 3: Clean Temporary and Session Files

    PHP session files and temporary files can also hold many inodes:

    # Delete session files older than 7 days
    find /tmp -name "sess_*" -type f -mtime +7 -delete
    
    # Delete old PHP temporary files
    find /tmp -name "*.tmp" -type f -mtime +7 -delete
    
    # Delete inactive cache files
    find /home/username/public_html/tmp -type f -mtime +30 -delete

    Solution 4: Clean Old Backups and Archives

    Unneeded backup files also hold inodes:

    # Find large backup files
    find /home/username -name "*.bak" -o -name "*.backup" -o -name "*.tar.gz" -o -name "*.zip" | head -20
    
    # Delete old backups (older than 30 days)
    find /home/username -name "*.bak" -mtime +30 -delete
    find /home/username -name "*.backup" -mtime +30 -delete
    find /home/username -name "*.tar.gz" -mtime +30 -delete

    Solution 5: Clean cPanel and WHM Cache

    cPanel and WHM also store cache files that can hold inodes:

    # Clean cPanel cache
    rm -rf /home/username/.cpanel/cache/*
    
    # Clean ea4 cache (if exists)
    rm -rf /home/username/.cpanel/ea4/cache/*
    
    # Clean old cPanel logs
    find /home/username/logs -name "*.log" -mtime +30 -delete

    Solution 6: Configure Logrotate

    After successfully fixing inode exhaustion, the next step is preventing it from happening again. Configure logrotate for all log files:

    # Create logrotate configuration for error_log
    cat > /etc/logrotate.d/error_log << 'EOF'
    /home/*/public_html/error_log {
        daily
        missingok
        rotate 7
        compress
        delaycompress
        notifempty
        create 644 nobody nobody
    }
    EOF

    Or if you're using cPanel, you can use cPanel's built-in log rotation feature → cPanel → Metrics → Metrics → Errors.

    Pro Tips: Preventing Inode Exhaustion

    1. Monitor inode usage regularly — set up a cron job for alerts: 0 */6 * * * USAGE=$(df -i / | tail -1 | awk '{print $5}' | tr -d '%'); [ $USAGE -gt 80 ] && echo "ALERT: Inode usage at $USAGE%" | mail -s "Inode Alert" admin@domain.com
    2. Use logrotate for all log files — make sure all actively written logs are configured with logrotate. Especially error_log, access_log, and application log files.
    3. Clean cache regularly — set up a cron job to delete old cache files: 0 2 * * * find /home/*/public_html/wp-content/cache -type f -mtime +7 -delete
    4. Limit files per directory — some applications like email or file storage can generate thousands of small files. Consider limiting the number of files or using alternative storage.
    5. Use tmpfs for temporary files — mount /tmp as tmpfs (RAM-based) for temporary files. Files on tmpfs don't consume inodes on the main filesystem.
    6. Set inode quota per account — if you manage a cPanel server, consider setting per-account inode quotas to prevent one account from consuming all server inodes.
    7. Audit files regularly — at least once a month, check which directories are holding the most files and clean up what's not needed.

    FAQ

    How do I increase inode quota on a cPanel server?

    Inode quota is usually set by the hosting provider and can't be changed by the user. If you manage your own server (VPS/dedicated), you can increase inode quota by creating a new filesystem and mounting it to the directory that needs more inodes. But the better approach is to reduce unnecessary files — because adding inodes means adding a new filesystem that also requires additional management.

    Is it safe to delete error_log?

    Yes, deleting error_log is generally safe. Error_log only contains debugging information and errors that have already occurred. After errors are fixed, old error_log files are no longer needed. But make sure you're not currently in the process of debugging a specific issue — because error_log might contain clues needed to find the root cause.

    Why does df -h show space is still free but I can't create new files?

    Because the problem isn't disk space, but inodes. df -h shows disk space usage (size in bytes). To check inodes, use df -i. If df -i shows IUse% = 100%, then inodes are exhausted even though space is still available.

    How many inodes are safe for a cPanel server?

    As a rule of thumb, keep inode usage below 80%. If it's approaching 90%, do an audit and cleanup immediately. For shared hosting servers with many accounts, consider setting per-account inode quotas (e.g., 250,000 per account) to prevent one account from consuming all server inodes.

    Related Issues

    Conclusion

    Inode exhaustion is a problem that's often ignored until it happens — and when it does, the impact can be severe: all websites down, services dead, and you can't even delete files for recovery. With df -i and find . -name 'error_log' -exec du -ch {} + | grep total, you can detect this problem before it's too late.

    Most importantly, prevention is always better than treatment. Configure logrotate, monitor inode usage regularly, and clean cache on a regular basis. Don't wait until inodes run out and all websites go down — trust me, I've been through it at 3 AM and don't want to go through it again.