Troubleshoot High Softirq CPU: Step-by-Step Guide 2026

Skip the fluff. You’re here because top is showing %si pegged at 50%+ on one or two cores, ksoftirqd/0 is sitting near the top of the process list, and your application is doing nothing. Been there, fixed it, more than once. Here’s the exact sequence I run — in order — whenever a high softirq CPU ticket lands on my desk.

This isn’t a theory post. Everything below is the drill I actually follow on Linux boxes (Ubuntu, Debian, and RHEL-based distros all behave the same way here) when a client’s server starts burning CPU doing nothing useful. Work through these steps in order and you’ll either find the cause or eliminate most of the usual suspects in under an hour. If you haven’t spent time reading load average properly, start with reading Linux load average the right way — it gives you the baseline vocabulary for the rest of this.

First, Let’s Agree on What We’re Looking At

High softirq CPU is not the same as the server being busy. It’s the kernel itself drowning in deferred interrupt work, and it usually shows up as a lopsided picture: total CPU load looks scary, but %usr (user space — your actual application) is quiet while %si climbs and ksoftirqd threads burn cycles. The impact on production is sneaky. Latency creeps up, throughput plateaus, half-open TCP connections start piling up, and users report the site is slow even though the webserver has headroom to spare. Let it run long enough and the cheap fix (reboot) stops working — you’re just gambling on timing at that point.

Quick primer if you haven’t dug into this before. Hardirq is the interrupt handler that runs the moment an interrupt arrives; it’s supposed to be short and snappy. Softirq is the deferred part — the kernel queues the heavy lifting and processes it later, when the core is free. When the queue overflows, the kernel spins up ksoftirqd, a dedicated thread that churns through the backlog on CPU time of its own. So if ksoftirqd is high, you’re not looking at a busy application. You’re looking at a kernel that can’t keep up with interrupt work. On virtual machines this can overlap with hypervisor-side noise, so if your box is a VPS it’s worth also checking CPU steal on Proxmox VPS before you blame the guest.

From the tickets I’ve handled, the causes cluster into a handful of buckets. (1) Interrupt storms from a single NIC — not necessarily huge bandwidth, but a brutal packet-per-second rate. (2) A single-queue NIC, or a multi-queue NIC with RSS disabled, dumping every packet onto one core — especially when irqbalance is off. (3) GRO/GSO/TSO offloads silently disabled, inherited from an old config or lost after a driver change. (4) Container traffic (veth, docker0, OpenVZ) hammering NET_RX and SCHED softirqs. And (5) plain old kernel or driver bugs that only a version bump fixes. The good news: all of these show up in the same diagnostic trail, so let’s walk it together.

Step 1: Confirm It’s Softirq, Not Something Else

Don’t tune anything yet. Confirm the diagnosis first. Two commands, ten seconds:

mpstat -P ALL 1

Watch the %si column. Consistently above 30-40% on any core, climbing while %usr stays flat? That’s your smoking gun. Then check the process list:

top -d 1

If ksoftirqd/x is floating near the top with serious CPU usage, we’re on the right track. If %usr or %sys is what’s pegged instead, this isn’t a softirq story — go look at your application or your syscalls.

Step 2: Read /proc/softirqs — the Smoking Gun

This file is your best friend for softirq work. It breaks down counts per vector and per CPU, so you can see exactly which type of softirq is exploding:

cat /proc/softirqs

Watch it over time to catch the trend:

watch -n1 cat /proc/softirqs

Look at NET_RX per core. If it’s piled onto one or two cores while the others sit idle, the interrupt isn’t being distributed. That’s your lead for the RPS and IRQ affinity fixes later in this guide.

Step 3: /proc/interrupts and irqtop — Who’s Interrupting?

Next, find out who’s actually generating the interrupts:

cat /proc/interrupts

The rightmost column names the device. Look for your NIC (eth0, enp3s0, or similar PCIe name). If the count climbs by tens of thousands within seconds, that’s an interrupt storm. On kernel 5.14+, irqtop is easier to read:

irqtop

irqtop is basically top for /proc/interrupts — per-CPU interrupt counts in real time, sorted by activity. It’ll tell you which core is getting hammered and from which device.

Step 4: Which Interface Is Screaming?

Now identify the traffic source. Start with NIC counters:

ethtool -S eth0 | grep -iE "rx_packets|rx_missed|dropped|error"

A rx_missed counter climbing fast means the NIC itself is dropping packets because it can’t keep up. Then look at the traffic pattern:

iftop -i eth0 -n

And if you need packet-level detail, take a short sample:

tcpdump -i eth0 -n -c 100

If the screaming interface turns out to be a veth (docker0, vethxxxx), your story is containers — the fix is RPS on the veth side plus a hard look at your network design. Either way, remember: packet rate matters more than bandwidth here.

Step 5: Fix #1 — RPS and XPS to Spread the Load

RPS (Receive Packet Steering) is the answer when your NIC is single-queue — every inbound packet gets processed on one core, and that core drowns while the others idle. Check your queue count first:

ls /sys/class/net/eth0/queues/

Only rx-0? Single queue. Enable RPS for cores 0-3 (bitmask f in hex; cores 0-7 = ff, 0-15 = ffff):

echo f | sudo tee /sys/class/net/eth0/queues/rx-0/rps_cpus
echo 4096 | sudo tee /sys/class/net/eth0/queues/rx-0/rps_flow_cnt

Pair it with RFS (Receive Flow Steering) so a given flow always lands on the same core and keeps the CPU cache warm:

sysctl -w net.core.rps_sock_flow_entries=32768

One rule to remember: the total of rps_flow_cnt across all queues must not exceed rps_sock_flow_entries. And sysfs writes die on reboot — persist them in rc.local or a systemd unit if you want them permanent. For more sysctl-level tuning context, see our Linux kernel performance tuning post.

Step 6: Fix #2 — irqbalance and IRQ Affinity

If your NIC is multi-queue, RSS plus irqbalance is the cleaner path than RPS. First, is irqbalance even running?

systemctl status irqbalance

If it’s not, enable it:

systemctl enable --now irqbalance

It spreads interrupts across cores automatically. On big servers with heavy traffic I sometimes pin queues manually (queue 0 to cores 0-3, queue 1 to cores 4-7, and so on), but for most cases irqbalance plus RSS is enough. Start with irqbalance, measure, then go manual only if the numbers still look bad.

Step 7: Fix #3 — Turn NIC Offloads Back On

One of the most common silent killers: GRO/GSO/TSO offloads disabled, usually inherited from an old config or lost after a driver swap. With offloads off, the kernel does more work per packet and softirq climbs fast. Check the current state:

ethtool -k eth0

Look for these lines:

generic-receive-offload: on
generic-segmentation-offload: on
tcp-segmentation-offload: on

If anything is off, enable it:

ethtool -K eth0 gro on gso on tso on

GRO is the big one for NET_RX. Caveat: on bonded interfaces, VLANs, or flaky provider links, offloads can occasionally cause checksum or CRC errors — so keep an eye on ethtool -S and dmesg after flipping them.

Step 8: Fix #4 — Tune netdev_budget

ksoftirqd climbing is a sign the kernel is spending more time on softirq than its budget allows. That budget lives in net.core.netdev_budget (default 300) and net.core.netdev_budget_usecs. Give it more room when traffic is heavy:

sysctl -w net.core.netdev_budget=600
sysctl -w net.core.netdev_budget_usecs=8000

Persist it properly:

cat > /etc/sysctl.d/99-netdev.conf <<'EOF'
net.core.netdev_budget = 600
net.core.netdev_budget_usecs = 8000
EOF
sysctl --system

Not a silver bullet, but it frequently takes real pressure off ksoftirqd.

Step 9: Fix #5 — RSS Multi-Queue and Driver/Kernel Version

If your NIC supports multiple queues but only one is active, you're leaving performance on the table:

ethtool -l eth0

Check the Combined row. If it shows 1 but the max is 8, raise it:

ethtool -L eth0 combined 8

RSS then distributes packets across queues automatically — cleaner than RPS by far. If everything above still leaves you high, look at driver and kernel versions:

ethtool -i eth0
uname -r

Sometimes the honest answer is upgrading the kernel. Some netstack regressions are only fixed in newer releases. Test on staging first — this is production we're talking about.

Troubleshooting Cheat Sheet

Verification: Proof, Not Hope

After applying fixes, don't close the ticket yet. Watch for a few minutes:

mpstat -P ALL 1
watch -n1 cat /proc/softirqs

Success looks like this: %si drops, ksoftirqd disappears from the top of the process list, and NET_RX stops exploding. I also like to capture a before/after with sar:

sar -u -P ALL 1 5

On production, log what you changed — netstack changes always deserve a changelog entry. And to catch the next incident before your phone buzzes at 3 a.m., set up proper alerting — our Netdata server monitoring guide is a good starting point.

FAQ

That's the whole drill. Run it in order, change one variable at a time, and you'll find the culprit. Bookmark this post for the next time — because there will be a next time. And if you want the background reading, start with the load average article and the Netdata monitoring guide so the next incident catches itself before it reaches your users.