Migrate VPS KVM Storage to Fix Full Production Server Disk

Picture this: you’re sipping your morning coffee, everything’s running smooth, and then your monitoring dashboard lights up like a Christmas tree. Grafana alerts, Slack notifications, WhatsApp messages—all screaming that disk usage on your production server just hit 97%. You frantically SSH in and realize one VPS client’s disk ballooned from 40GB to 200GB overnight because they deployed Elasticsearch without any storage quota. And just like that, your entire production server is full. Every other VPS on that box starts throwing I/O errors. It’s like one tenant in an apartment building stuffing their room floor-to-ceiling, blocking the hallway for everyone else.

I’ve been there more times than I’d like to admit. One particular incident stands out: a production server hosting 28 KVM VPS instances suddenly had 6 of them throwing I/O errors because the underlying storage was completely full. The worst part? Several of those VPS instances hadn’t been actively used in months—the clients had stopped paying, but the VPS kept running and consuming disk space. That’s when I realized we had zero storage monitoring and no housekeeping procedures in place. In this article, we’ll walk through exactly how to sort and identify which VPS instances are eating up your storage, and how to migrate their disks without significant downtime. These aren’t theoretical steps—they’re the exact playbook I’ve used in production environments repeatedly.

Before we dive into the technical commands, it’s important to understand why this problem occurs in the first place. In a KVM infrastructure, each VPS has a disk image file (typically in .qcow2 or .raw format) stored on the production server’s storage. These files grow dynamically—meaning even if you allocate 100GB to a VPS, the disk image file doesn’t immediately consume 100GB on the host. But as data inside the VPS grows, so does the disk image file. Without proper monitoring and housekeeping, your production server’s storage fills up before you even notice. This is a classic problem in managed VPS hosting environments that NOC engineers worldwide face regularly.

Why Does Production Server Storage Fill Up?

Before we sort and migrate, let’s understand the root causes. Production server storage fills up for several primary reasons:

• VPS disks growing without limits — Clients deploy applications that constantly generate data (logs, databases, caches). Without quotas, disks keep expanding until they consume all available server space.
• Dormant VPS still consuming space — VPS instances that clients no longer use, but haven’t been destroyed. Their disk images are still sitting on storage.
• Snapshot accumulation — If you use snapshots for backups, each snapshot consumes additional space. Many NOC engineers forget to clean up old snapshots.
• Unrotated log files — VPS running applications with high logging (Elasticsearch, Apache, Nginx access logs) can generate tens of GBs of logs per day.
• Temporary files and caches — /tmp, /var/cache, and application cache directories that are never cleaned.

Warning Signs You Shouldn’t Ignore

Storage problems rarely appear without any warning. Here are several signs you should watch for before the server actually crashes:

• I/O wait spikes dramatically in monitoring (CPU steal time can also increase when storage becomes a bottleneck)
• Specific VPS instances start responding slowly, even though bandwidth and CPU look normal
• “No space left on device” error messages appearing on the host OS
• Write errors on multiple VPS instances simultaneously
• Backup processes failing because there’s not enough space to write backup files

Step 1: Identify and Sort VPS by Disk Usage

Alright, let’s start with the most fundamental step: identifying which VPS instances are consuming the most storage. When you’re hosting multiple VPS on a single server, you can’t check them one by one—that would take forever. You need a fast and efficient approach.

First, SSH into your production server as root:

ssh root@203.0.113.10

Then run the following command to see all running VPS instances along with their disk usage:

for vm in $(virsh list --name); do echo "=== $vm ==="; virsh domblklist $vm 2>/dev/null | grep -E '.(qcow2|raw|img)$' | while read dev disk; do if [ -f "$disk" ]; then echo "  Disk: $disk"; echo "  Size: $(du -h $disk | awk '{print $1}')"; echo "  Alloc: $(qemu-img info $disk 2>/dev/null | grep 'disk size' | awk '{print $3, $4}')"; fi; done; done

The command above loops through all VPS instances, checks their disk image files, and displays the actual size. The output looks something like this:

=== vps-client-a ===
  Disk: /var/lib/libvirt/images/vps-client-a.qcow2
  Size: 45G
  Alloc: 42.3 GB

=== vps-client-b ===
  Disk: /var/lib/libvirt/images/vps-client-b.qcow2
  Size: 200G
  Alloc: 187.6 GB

=== vps-client-c ===
  Disk: /var/lib/libvirt/images/vps-client-c.qcow2
  Size: 10G
  Alloc: 3.2 GB

=== vps-client-d ===
  Disk: /var/lib/libvirt/images/vps-client-d.qcow2
  Size: 80G
  Alloc: 76.1 GB

See how vps-client-b is consuming nearly 188GB? On a server with 1TB of total storage hosting 28 VPS instances, one VPS eating 188GB is almost 20% of your total capacity. And that’s before accounting for other large VPS instances.

For a faster sorted view, use this one-liner:

for vm in $(virsh list --name); do disk=$(virsh domblklist $vm 2>/dev/null | grep -E '.(qcow2|raw|img)$' | awk '{print $2}'); if [ -f "$disk" ]; then size=$(du -b "$disk" | awk '{print $1}'); echo "$size $vm $disk"; fi; done | sort -rn | head -20

This command sorts VPS instances from largest to smallest. Super practical when you have many VPS instances on a single server.

Step 2: Deep Analysis — What’s Making the VPS Disk So Large?

After identifying which VPS is the biggest offender, the next step is to log into that VPS and figure out what’s causing the disk bloat. SSH into the problematic VPS:

virsh console vps-client-b

Or connect via SSH to the VPS IP address. Then run disk analysis:

df -h
du -sh /* 2>/dev/null | sort -rh | head -15

This shows you which directories are consuming the most space. Common culprits:

/var/log         45G
/var/lib/mysql   38G
/home/user       67G
/tmp             12G

If /var/log is huge, log rotation probably isn’t running. Check with:

ls -lah /var/log/*.log* | head -20
ls -lah /var/log/syslog* | head -10

If /var/lib/mysql is large, the database has grown significantly and might need optimization or offloading to separate storage.

Step 3: Create a Migration Plan

Now you know which VPS is problematic and what’s causing it. You have several options:

1. In-place cleanup — Clean logs, cache, and unnecessary files inside the VPS without migration.
2. Disk image shrink — After cleaning files inside the VPS, you can shrink the disk image to reclaim space.
3. Migrate to different storage — Move the VPS disk image to additional storage (NAS, SAN, or another server).
4. Migrate VPS to another server — Move the entire VPS to a production server with more storage available.

In this article, we’ll focus on options 3 and 4—migration. Why? Because sometimes cleanup alone isn’t enough. The VPS genuinely needs more storage, or it’s time to offload some VPS instances from the production server.

Step 4: Migration Preparation

Before starting the migration, make sure these prerequisites are in place:

• Destination storage is ready — Whether it’s NAS (NFS mount), another storage server, or a new LVM volume. Verify sufficient space.
• Network bandwidth is adequate — Migrating large disk images requires sufficient bandwidth. Use a dedicated network (management VLAN) if possible.
• Backup has been performed — ALWAYS backup first. This is non-negotiable.
• Schedule a maintenance window — Notify clients about downtime. Never migrate silently—it’s unprofessional and erodes trust.
• Document every step — Log everything in your ticketing system. You don’t want to forget what was done if issues arise later.

du -sh /var/lib/libvirt/images/vps-client-b.qcow2
# Output: 200G

# Check space on destination
df -h /mnt/nas-storage/
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/nfs-share  2.0T  800G  1.2T  40% /mnt/nas-storage

Step 5: Migrate VPS Disk with virsh

There are several migration methods you can choose from depending on your situation:

Method A: Offline Migration (Blockcopy via SCP/rsync)

This is the most straightforward approach. Shut down the VPS, copy the disk image file, and restart the VPS at the new location.

# 1. Shut down the VPS
virsh shutdown vps-client-b
# Wait until the VPS is completely off
virsh list --name | grep vps-client-b
# If no output, it's shut down

# 2. Verify the disk file
ls -lah /var/lib/libvirt/images/vps-client-b.qcow2
# -rw------- 1 root root 200G Jul 15 10:30 /var/lib/libvirt/images/vps-client-b.qcow2

# 3. Copy to destination storage
rsync -avP --progress /var/lib/libvirt/images/vps-client-b.qcow2 /mnt/nas-storage/vps-client-b.qcow2

# 4. Verify file integrity at destination
md5sum /var/lib/libvirt/images/vps-client-b.qcow2
md5sum /mnt/nas-storage/vps-client-b.qcow2
# MD5 hashes MUST match

# 5. Update XML config to point to new location
virsh dumpxml vps-client-b > /tmp/vps-client-b.xml
# Edit the XML path:
# Change /var/lib/libvirt/images/vps-client-b.qcow2
# To /mnt/nas-storage/vps-client-b.qcow2

# 6. Redefine VPS with new config
virsh undefine vps-client-b --nvram
virsh define /tmp/vps-client-b.xml

# 7. Start the VPS
virsh start vps-client-b

Method B: Live Migration (virsh migrate)

If you can’t tolerate any downtime at all, live migration is the way to go. But keep in mind—live migration is typically used to migrate between servers, not between storage locations on the same server.

# Live migration to another server
virsh migrate --live vps-client-b qemu+ssh://server-dest/system

# Check migration status
virsh domjobinfo vps-client-b

Live migration works great when you want to move a VPS from a storage-full production server to a new server with more available space. But make sure your inter-server network is fast enough—ideally 1Gbps dedicated, with 10Gbps being even better.

Method C: Block Copy (virsh blockcopy)

This is a more advanced approach, suitable for live block device migration:

# Start block copy to a new file
virsh blockcopy vps-client-b vda /mnt/nas-storage/vps-client-b-copy.qcow2 --wait --verbose

# After completion, pivot the block device
virsh blockjob vps-client-b vda --pivot

# Verify
virsh domblkinfo vps-client-b vda

Troubleshooting: Common Migration Issues

Pro Tips from the Field

# Quick alert script - add to crontab
#!/bin/bash
USAGE=$(df /var/lib/libvirt/images | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $USAGE -gt 80 ]; then
    echo "WARNING: Storage usage at ${USAGE}% on $(hostname)" | mail -s "Storage Alert" noc@company.com
fi

Related Issues

Storage problems on production servers are often closely related to several other issues you should understand. For example, if you’re experiencing how to check disk full on a VPS, that could be an early warning before things get worse. Additionally, understanding VPS Linux storage optimization can help prevent similar problems in the future. And if you’ve determined it’s time to upgrade your infrastructure, check out our guide on migrating VPS to a new server for a more comprehensive walkthrough.

Wrapping Up

Full storage on a production server is a serious problem, but it’s not the end of the world. With a structured approach—identification, analysis, planning, and proper migration execution—you can resolve this without losing data or your clients’ trust. The key takeaway: don’t wait until the server is full to take action. Set up monitoring, establish housekeeping routines, and always have a backup plan for every VPS you manage. If you need further help with VPS infrastructure management, don’t hesitate to reach out to an experienced NOC team.