Ahosting Logo
Knowledge Base

Understanding Load Average and CPU Steal

Load average counts more than CPU workWhat it counts· processes running· plus processes waiting· and on Linux, waiting includes waiting for diskSo a high load can mean· a busy CPU· or a slow disk with the CPU nearly idle· which are completely different problemsThe steal figureTime your virtual machine wanted the CPU and did not get it, because the physical host was busy.Persistently high steal is a host problem, not yours.

Load average appears in every monitoring tool and is one of the most misread numbers in server administration. The same value can mean a healthy machine or a badly struggling one.

What it actually counts

uptime
cat /proc/loadavg

Three numbers: the average over one, five and fifteen minutes.

The common description, "how busy the processor is", is wrong on Linux. Load counts processes that are running plus processes that are waiting, and the waiting includes processes blocked on disk.

That single detail explains a situation people find baffling: a machine with a load of 12 and processors that are nearly idle. Nothing is computing. Everything is waiting for storage.

Judge it against the core count

nproc

A load of 4 on a four-core machine means it is fully occupied with nothing queueing; the ideal state, not a problem.

The same load of 4 on a single-core machine means three processes are waiting at any moment, and everything feels slow.

So the useful figure is load divided by cores. Around 1.0 is fully used. Consistently above 2.0 means work is queueing faster than it clears.

Read the direction, not the reading

The three numbers exist to show a trend.

First number much higher than the third: something started recently. A traffic spike, a backup, a crawler.

All three similar and high: a sustained condition, which is the one that needs a decision rather than patience.

First much lower than the third: it is already recovering, and whatever you were about to do is probably unnecessary.

Steal: the column that is not your fault

top

On the CPU line, past us, sy, id and wa, there is st: steal.

Steal is time your virtual machine was ready to run and the physical host gave the processor to another guest instead. Your server was not busy. It was not allowed to work.

Occasional low single digits are normal on shared virtualisation. Consistently above a few percent means the host is oversubscribed, and no amount of optimisation inside your server will change it. That is a conversation with the provider, and it is worth having with figures rather than impressions.

This is the single most useful thing in top for anyone on a VPS, and almost nobody looks at it.

Separating the causes

wa high: waiting on disk. Storage is the constraint.

us high: your own processes are genuinely computing. Find which:

ps aux --sort=-%cpu | head
top -o %CPU

sy high; the kernel is busy, often with network or context switching.

st high; the host, as above.

Answering that first question changes everything you do next, and it takes ten seconds.

Load with idle processors

The classic pattern: load 15, processors mostly idle, everything slow.

iostat -x 2 5

If a device is near 100% utilised with a growing queue, storage is the bottleneck. On a VPS that can be your own I/O, or a neighbour's on shared storage, and the two look identical from inside.

The other cause of this pattern is memory pressure: processes waiting on swap look exactly like processes waiting on disk, because they are. Configuring swap and memory deals with it.

What to do about a genuinely high load

Find the process before changing anything. The usual answers on a web server are a database query with no index, a crawler being served uncached pages, or a backup running at the wrong hour.

All three are cheaper to fix than to outgrow. Adding cores to a server running one bad query buys a slightly slower arrival at the same place, managing VPS resources goes over the wider picture, and scaling and upgrading goes into when growth genuinely is the answer.

Where the load figure comes from

Reading the raw value occasionally helps, because it shows the queue length directly instead of the smoothed averages.

cat /proc/loadavg
ps -eo state,pid,comm | awk '$1 ~ /^[RD]/' | head -20

The fourth field of the first command is running processes over total processes. The second lists what is actually in those states: R is running, D is uninterruptible sleep, which almost always means waiting on storage.

A list full of D processes is the direct evidence that the load is disk-driven rather than processor-driven, which is the distinction the averages cannot make.

Containers report the host, not themselves

Worth knowing because it produces alarming and meaningless numbers.

Inside a container, the load average is typically the host machine's, including work done by every other container on it. A container doing almost nothing can show a load of 40.

nproc
cat /sys/fs/cgroup/cpu.max 2>/dev/null
cat /sys/fs/cgroup/cpu.stat 2>/dev/null | head -5

The cgroup figures describe what this container may actually use, and the throttling counters there are the meaningful signal. A rising throttled count means the container is hitting its own limit, whatever the load average says.

The same caution applies to memory readings inside a container, which is why capacity decisions made from free alone are frequently wrong.

Correlate the load with what happened

A load figure alone says the machine is busy. It does not say why, and the answer is usually in what else occurred at that time.

sar -q -f /var/log/sa/sa$(date +%d) | tail -20
awk '{print $4}' ~/logs/example.com | cut -d: -f2 | sort | uniq -c

Comparing the load by hour against requests by hour separates the two common cases immediately: load that tracks traffic is capacity, and load that spikes at a fixed hour regardless of traffic is a scheduled job.

The second is much more common than people expect, and it is cheaper to fix, moving a backup or an import to a quieter hour costs nothing. Managing VPS resources goes into keeping that history.

Steal is worth documenting before raising it

A conversation with a provider about oversubscription goes better with figures than with an impression.

vmstat 5 60 | awk 'NR>2 {print $NF}' | sort -n | tail -5
sar -u 1 60 | tail -3

Collect steal readings over a period instead of a moment, note the times of day, and compare a busy hour against a quiet one.

Sustained steal of several percent, documented across days, is a specific and answerable complaint. A single reading is not, and it is the difference between being moved to a less crowded host and being told it is normal. Scaling and upgrading your VPS explains when the answer really is a larger plan.