Detect Deleted Files on Linux with lsof +L1

Introduction

Have you ever encountered a situation where your server’s disk space is full, you check every directory with du -sh /*, but the results don’t reflect the actual disk usage? For example, df -h shows the disk is 95% full, but when you add up all the directories, it only totals 60%. Where’s the other 35%?

The first time I hit this problem, I was confused for almost an hour. I’d checked logs, temporary directories, even cache files — none of them were large enough to explain that 35% gap. It wasn’t until I used lsof +L1 that I found the secret: hundreds of files that had been deleted from the filesystem but were still held open by running processes. These files were no longer visible in any directory, but their space was still being counted by the system.

In this article I’ll show you how to find these “ghost” files, understand why deleted files don’t release their space, and most importantly — how to safely free that space. Every command below I’ve used many times on production servers and they’re proven effective.

Symptoms of Disk Space Full But Files Not Visible

Before getting into troubleshooting commands, it’s important to know the symptoms that indicate deleted files are still holding disk space:

• Discrepancy between df -h and du -sh — the most classic symptom. df shows the disk nearly full, but du in all directories doesn’t add up to the same number.
• Disk space decreases on its own without new files — sometimes space gradually shrinks even though you haven’t uploaded or created new files. This could be because logs keep growing but old processes are still holding rotated log files.
• Backup fails with “No space left on device” — this error appears even though logically there should be enough space. Because the space is being held by files that are no longer visible.
• Specific process uses excessive RAM — some processes holding file descriptors can also use extra RAM for buffering, especially if those files are very large.
• Log rotation isn’t effective — you’ve rotated logs, old log files are deleted from the directory, but space doesn’t return. This is because the logging process is still writing to the same file descriptor.

Root Cause: Why Can Files Be Deleted But Space Not Return?

To understand this problem, we need to know how Linux handles files that are currently open by a process.

1. The Concept of File Descriptors in Linux

When a process (such as Apache, PHP-FPM, or MySQL) opens a file, the operating system creates a file descriptor — a “link” between that process and the file on the filesystem. A file descriptor is like a key held by the process as long as the file remains open.

2. What Happens When a File Is Deleted?

When you run rm /path/to/file, here’s what happens:

• The file name is removed from the directory (inode link count decreases)
• But if a process still has the file open, the file descriptor remains active
• The filesystem marks the file as “deleted” but doesn’t free the space until all file descriptors to that file are closed

So the file is no longer visible in ls, but its space is still being used by the process holding that file descriptor. This is why you see the discrepancy between df and du.

3. Why Do Processes Still Hold the File?

There are several reasons why a process still holds a file descriptor even after the file is deleted:

• Imperfect log rotation — the logging process needs to be restarted after logs are rotated. If not restarted, the process continues writing to the same file descriptor (the old deleted file).
• Stuck PHP-FPM workers — PHP workers that have finished but haven’t been restarted can still hold file descriptors to logs or temporary files.
• Slow cron jobs — cron scripts processing large files that haven’t finished will hold file descriptors until completion.
• Long-running batch processes — database imports/exports, backups, or other batch processes that take a long time.

Command #1: Find All Deleted Files Still Held by Processes

The first step is finding which files have been deleted but are still holding space. We use lsof with the +L1 flag, which means “link count less than 1” — files that no longer have a directory entry but are still held by a process.

lsof +L1 | grep deleted | sort -k 7 -rn | head -10

Let’s break down this command one by one:

Breakdown of Each Part

1. lsof +L1 — lsof (list open files) is a command to view all files currently opened by processes on the system. The +L1 flag filters only files with a link count less than 1 — files that have been deleted from the directory (link count is 0) but are still held by a process. Without this flag, lsof would show all open files, including those still in directories.

2. | grep deleted — Filters the lsof output to show only lines containing the word deleted. In lsof output, deleted files are marked with the status (deleted) at the end of the line. This ensures we only see files that have been deleted but are still holding space.

3. | sort -k 7 -rn — Sorts results by column 7 numerically in reverse (-rn means reverse numeric). Column 7 in lsof output is the file size in bytes. So the largest files appear first — helping us prioritize which files are holding the most space.

4. | head -10 — Shows only the first 10 lines — the 10 largest files still holding space. Usually these top 10 files are enough to explain a significant space discrepancy.

Example Output

COMMAND     PID   USER   FD   TYPE DEVICE SIZE/OFF   NLINK NODE NAME
lsphp      8834  nobody    5w  REG  253,1  12582944      0  456 /var/log/lsphp80/error.log-deleted (deleted)
lsphp      8834  nobody    6w  REG  253,1   8388608      0  457 /var/log/lsphp80/access.log-deleted (deleted)
lsphp      9012  nobody    5w  REG  253,1   6291456      0  458 /var/log/lsphp80/error.log-deleted (deleted)
mysqld     1205  mysql     4w  REG  253,1   5242880      0  612 /var/log/mysql/slow.log-deleted (deleted)
lsphp      8834  nobody    7w  REG  253,1   3145728      0  459 /tmp/session_phpxyz123-deleted (deleted)
nginx      1102  root      5w  REG  253,1   2097152      0  613 /var/log/nginx/access.log-deleted (deleted)

Interpreting the Output

From the output above, we can see several important things:

• Largest file — /var/log/lsphp80/error.log-deleted at 12 MB is still held by the lsphp process (PHP-FPM) with PID 8834. This file has been deleted from the directory but is still holding 12MB of space.
• Same pattern — there are 3 lsphp80/error.log-deleted files from different PIDs (8834 and 9012). This indicates multiple PHP-FPM workers are still holding file descriptors to old logs.
• Total space held — if you add them up, the top 10 files are holding about 40MB of space. But this is just the top 10 largest — there could be hundreds of smaller files also holding space.

Command #2: Identify Processes Holding Deleted Files

After finding which files are holding space, we need to know which processes are holding those files. This is important for determining next steps — whether to restart the process or just close the file descriptor.

lsof +L1 | grep deleted | awk '{print $1, $2, $7, $9}' | sort -k 3 -rn | head -20

This command displays: process name, PID, file size, and path of the deleted file. Results are sorted by largest file size.

Example Output

lsphp 8834 12582944 /var/log/lsphp80/error.log-deleted
lsphp 9012  8388608 /var/log/lsphp80/access.log-deleted
mysqld 1205  5242880 /var/log/mysql/slow.log-deleted
lsphp 8834  3145728 /tmp/session_phpxyz123-deleted
nginx 1102  2097152 /var/log/nginx/access.log-deleted

How to Read the Output

Each line shows:

• Column 1 (lsphp) — the name of the process holding the file descriptor. lsphp = LiteSpeed PHP (PHP-FPM), mysqld = MySQL/MariaDB, nginx = Nginx web server.
• Column 2 (8834) — the PID (Process ID) of that process. From this PID we can determine whether a restart is needed.
• Column 3 (12582944) — file size in bytes. To convert to MB, divide by 1,048,576.
• Column 4 (/var/log/…) — path of the file that was deleted but is still being held.

Command #3: Calculate Total Space Held

To find out how much total space is being held by deleted files, use this command:

lsof +L1 | grep deleted | awk '{sum += $7} END {printf "Total space held by deleted files: %.2f MBn", sum/1048576}'

Or the more detailed version broken down by process:

lsof +L1 | grep deleted | awk '{proc[$1]+=$7} END {for(p in proc) printf "%-15s %.2f MBn", p, proc[p]/1048576}' | sort -k2 -rn

Example Output

Total space held by deleted files: 847.32 MB

Or per process:

lsphp            612.45 MB
mysqld           156.23 MB
nginx             78.64 MB

From this output, it’s clear: the lsphp (PHP-FPM) process is holding over 600MB of space. This explains why the disk feels full even though files have been deleted from directories.

How to Free Space from Deleted Files

⚠️ SECURITY WARNING: Backup & Verify Before Truncating

Before we truncate file descriptors, you MUST understand that the truncate command permanently empties file contents. If that file turns out to still be needed (e.g., logs that haven’t been backed up), you’ll lose that data forever.

Backup files that can still be backed up:

# Backup log files still in the directory (if any)
cp /var/log/lsphp80/error.log /tmp/backup/error.log.bak-$(date +%Y%m%d-%H%M) 2>/dev/null

# Backup the full list of deleted files for reference
lsof +L1 | grep deleted > /tmp/deleted-files-list-$(date +%Y%m%d-%H%M).txt

Verify targets before truncating:

# Double-check which files will be truncated
lsof +L1 | grep deleted | grep lsphp

# Make sure no important files are caught in the process
# (e.g., database files, config files, or backup files)
lsof +L1 | grep deleted | grep -E "(mysql|postgres|config|backup)" 

Truncate command via /proc/PID/fd/FD:

The safest way to free space from deleted files is to overwrite the contents of the file descriptor. We can do this through /proc/PID/fd/FD:

# Truncate a specific file descriptor (replace PID and FD with values from lsof output)
echo -n "" > /proc/8834/fd/5
echo -n "" > /proc/8834/fd/6

This command will empty the file descriptor’s contents without closing the connection or stopping the process. After truncation, the process will continue writing to the same file descriptor but into the emptied space — meaning the space that was previously held will be returned to the system.

⚠️ WARNING: Don’t Run Truncate Commands Without Filtering

DO NOT run truncate commands for all file descriptors at once without verification. A command like this is extremely dangerous:

❌ DON'T DO THIS — could empty files that are still needed
lsof +L1 | grep deleted | awk '{print $2, $4}' | while read pid fd; do
  echo -n "" > /proc/$pid/fd/$fd
done

The command above will empty ALL file descriptors held by processes — including files that might still be needed (such as socket files, pipe files, or files actively being written to). The result can be fatal: process crashes, connections dropped, or data corruption.

Always: check which files will be truncated first, make sure only log or temporary files are being truncated, and never truncate file descriptors belonging to critical system processes.

Alternative: Restart the Process (Safest Method)

If you’re unsure or hesitant, the safest method is to restart the process holding the file. Once the process is restarted, all file descriptors are automatically closed and space returns:

# Restart PHP-FPM (will close all lsphp file descriptors)
systemctl restart lsphp

# Or for a specific PHP version
systemctl restart lsphp80

# Restart MariaDB (for MySQL log files)
systemctl restart mariadb

# Restart Nginx
systemctl restart nginx

⚠️ IMPORTANT: Restarting production processes must be done carefully. Make sure:

• There are no active requests being processed (or wait until they finish)
• No deployments or code changes are in progress
• Clients have been notified if there might be brief downtime
• Configuration backup exists before restarting

Verification after restart:

# Check if space has returned
df -h / | tail -1

# Check if file descriptors are clean
lsof +L1 | grep deleted | wc -l

Additional Command: Truncate All Deleted PHP-FPM Log Files

If you’re certain all deleted files held by lsphp are log files no longer needed, there’s a more efficient command to truncate them all at once:

gawk 'match($0, /lsphp.* (([0-9]+).* ([0-9]+)[uw].*deleted)/, a) {print "true > /proc/"a[2]"/fd/"a[3]}' <(lsof +L1 | grep deleted) | sh

Let's break down this command:

Breakdown of Each Part

1. gawk 'match($0, /lsphp.* (([0-9]+).* ([0-9]+)[uw].*deleted)/, a) — Uses gawk (GNU awk) with the match function to regex-match against each line of lsof output. This regex searches for lines containing lsphp (PHP-FPM process) and extracts: the process PID (a[2]) and file descriptor number (a[3]). The [uw] flag indicates the file descriptor is in write mode (w) or read-write mode (u).

2. {print "true > /proc/"a[2]"/fd/"a[3]} — For each match, prints a shell command that will empty that file descriptor. true > /proc/PID/fd/FD means writing the empty output of the true command to the file descriptor — this is lighter than echo -n "" because true doesn't allocate any buffer.

3. <(lsof +L1 | grep deleted) — Process substitution: runs lsof +L1 | grep deleted and supplies its output as an input file for gawk.

4. | sh — Pipes all the printed truncate commands to the shell for execution. Each line runs a truncate for one file descriptor.

⚠️ Strong Warning for This Command

The command above will only capture files starting with "lsphp" thanks to the lsphp regex filter. This is safer than truncating all files, but still needs attention:

• Make sure there are no non-log files starting with "lsphp" — for example, if there's an important temporary PHP file, this command will empty it too.
• Verify the output first — before piping to sh, check what will be executed by removing | sh from the end of the command.
• Don't run on production without testing first — try it in a staging environment first to make sure there are no side effects.

Pro Tips: Preventing Deleted Files from Holding Space

1. Restart PHP-FPM after log rotation — this is the most important step. If you rotate PHP-FPM logs, make sure to restart the PHP-FPM process so old file descriptors are closed and writing starts on the new log file. Add to crontab: 0 0 * * * /usr/bin/systemctl restart lsphp80
2. Monitor disk space regularly — set up a cron job to alert when the discrepancy between df and du exceeds 10%: 0 */6 * * * DIFF=$(($(df / --output=pcent | tail -1 | tr -dc '0-9') - $(du -s / 2>/dev/null | awk '{print int($1/1024/1024/100*100)}'))); [ $DIFF -gt 10 ] && echo "ALERT: Disk space discrepancy - $DIFF%" | mail -s "Disk Alert" admin@domain.com
3. Use logrotate with copytruncate — in /etc/logrotate.d/lsphp, add the copytruncate option. This option creates a copy of the old log file then empties the original (instead of deleting and creating a new one). This way, the logging process continues writing to the same file descriptor without needing a restart.
4. Set up logrotate for all service logs — make sure all actively written logs (Apache, Nginx, PHP, MySQL, mail) are configured with logrotate using copytruncate or followed by a service restart.
5. Check deleted files regularly — add this command to your monitoring script: lsof +L1 | grep deleted | wc -l | awk '{if ($1 > 50) print "ALERT: "$1" deleted files still held by processes"}'
6. Use tmpfs for temporary files — for temporary files that are frequently created and deleted (like PHP session files), use tmpfs mounted in RAM. The space won't affect the main disk.
7. Set ulimit for file descriptors — limit the number of file descriptors per process to prevent one process from holding too many files. Add to /etc/security/limits.conf: nobody soft nofile 65535 and nobody hard nofile 65535

FAQ

Will deleted files holding space eventually go away on their own?

No. Deleted files still held by processes will continue holding space until that process is closed, restarted, or the file descriptor is manually closed. There's no timeout or auto-cleanup from the operating system. If you leave it alone, that space will be held for as long as the process is running — potentially for days or weeks.

How do I know if a file held by a process is a log or some other important file?

Look at the file path in the lsof output. If the path is in /var/log/, it's likely a log file. If in /tmp/, it's a temporary file. If in /var/lib/mysql/, it's a database file (be careful!). If you're unsure, don't truncate — just restart the process. Restarting is safer than blind truncation.

Is it safe to run the gawk truncate command on production?

Relatively safe since the lsphp filter only captures PHP-FPM files. But there's still a risk: if there's an important temporary PHP file (such as an upload being processed), that file will also be truncated. Best to check the output first without | sh, then run it once you're confident.

Why should I use lsof +L1 instead of just lsof | grep deleted?

lsof +L1 filters by link count — only files that no longer have links in the directory (link count < 1). Meanwhile lsof | grep deleted just searches for the word "deleted" in the output. Both can give different results. lsof +L1 is more accurate for finding files that have truly been deleted from the directory but are still holding space. Combining both (lsof +L1 | grep deleted) gives the most precise results.

Related Issues

• How to Troubleshoot High Database Load in MariaDB — MySQL log files that are deleted but still held can also cause disk space to fill up
• How to Detect & Delete Email Spam on cPanel Server — spam emails piling up in the queue can also consume significant disk space
• How to Fix Apache Slots Being Full (503 Service Unavailable) — Apache slots filling up often happens alongside disk space issues due to constantly growing logs

Conclusion

Deleted files that still hold disk space is a problem that often seems trivial but can be fatal if not handled. With lsof +L1, you can find these "ghost" files quickly. The two core commands you need to memorize are:

• lsof +L1 | grep deleted | sort -k 7 -rn | head -10 — to find the largest deleted files still holding space
• echo -n "" > /proc/PID/fd/FD — to free space from a specific file descriptor (after verification!)

Most importantly, prevention is always better than treatment. Make sure log rotation is followed by service restarts, monitor disk space regularly, and always verify before truncating. Better to prevent than to have to explain to a client why their server disk is full even though log files have been deleted — trust me, I've been there and don't want to go through it again.