Ahosting Logo

Dedicated Server

Monitoring Performance and Server Health

Three hardware failures that produce no visible symptomNothing slows down, nothing errors, nothing tells youA failed array membersilent by design; that is whatredundancy is forCorrectable memory errorscorrected until they are notA failed redundant power supplythe server runs on the otherone, happilyWhat catches themarray status, SMART, and themanagement controller alertsWhy it matterseach one has removed yourredundancy without telling youThe restordinary resource monitoring,same as any serverIn all three cases the machine keeps working perfectly, which is exactly the problem: you are now one failure from anoutage and do not know it.

The generic side of server monitoring, load, memory, disk, processes, works the same on a dedicated server as anywhere else, and managing VPS resources and monitoring performance goes into it.

What is different here is the hardware. On a virtual machine somebody else watches it. On your own server, three failure modes produce no error at all, and each is invisible to every tool that reads the operating system.

A failed member in a redundant array

This one is silent by design. Redundancy exists so that a disk failure changes nothing the operating system can see. The array keeps serving at full function.

What it no longer has is protection. A degraded array is not an incident. It is a countdown, and the clock started at a moment nobody observed.

cat /proc/mdstat
storcli /c0 show
megacli -LDInfo -Lall -aALL

The command depends on the controller. What matters is that it runs on a schedule and its output reaches you, because a person who has to remember to check it will not.

Alert on any state other than optimal, and treat a rebuild as an event worth watching too, rebuilds put sustained load on the remaining disks, which is when a second, already-marginal disk tends to fail.

Setting up RAID and storage management deals with the arrangement itself.

Correctable memory errors

Server memory corrects single-bit errors transparently. The machine carries on, entirely normal.

A module that is beginning to fail produces a climbing count of these corrections long before it produces an uncorrectable error, and the uncorrectable one crashes the machine.

edac-util -v
ras-mc-ctl --summary
dmesg | grep -i -E 'edac|mce|hardware error'

What matters is the trend instead of the number. A few over a year is unremarkable; a count that is rising week by week is a module telling you it will fail, with enough notice to replace it during a maintenance window instead of at three in the morning.

This is the single most valuable thing to monitor on a dedicated server, and almost nobody does.

Thermal throttling

A processor that gets too hot reduces its own speed to protect itself. The machine stays up. Everything simply becomes slower.

The symptom is a server that performs worse than it used to with no change in load, and it is diagnosed as an application problem far more often than as a cooling one, because nothing reports a fault.

ipmitool sdr type temperature
ipmitool sdr type fan
sensors

Causes are dull and physical: a failed fan, dust, or a rack that has become warmer than it was. All are cheap to fix and none announces itself.

Using IPMI and out-of-band management goes over reaching these readings even when the operating system is unresponsive, which is the other reason that interface matters.

Disk health before failure

Individual drives report their own condition, and a drive that is deteriorating usually says so first.

smartctl -a /dev/sda
smartctl -H /dev/sda

The attributes worth alerting on are reallocated sectors, pending sectors, and any figure that is growing. An overall assessment of "PASSED" alongside a rising reallocated count is not reassurance: it is a drive on its way out that has not yet crossed the threshold.

Behind a hardware array controller, the operating system may not see the drives directly, and the controller's own tool is where these attributes live instead.

Alert, do not display

A dashboard reports the present. Hardware failures happen while nobody is looking at the present.

Everything above should page you rather than appear on a screen: array not optimal, correctable errors rising, a temperature or fan reading out of range, a drive attribute growing.

And send those alerts through something that does not depend on the server. A machine with a failing array cannot reliably tell you about its failing array.

Test that the alert works

An untested alert is a belief. The failure mode is uniform: it was configured, it was never triggered, and it did not fire when it mattered.

Trigger each one deliberately once, fill a test file to cross a disk threshold, stop a monitored service, use the tool's own test facility. Confirm the message arrives, on the device you expect, out of hours.

Then do it again after any change to the alerting path, because that is precisely when it silently stops working.

Know what normal looks like

None of this is useful without a baseline. Record what the machine's temperatures, error counts and load look like during a week you consider healthy.

Hardware problems announce themselves as deviations from that record long before they announce themselves as failures. Troubleshooting common dedicated server issues explains acting on what you find.

And when the hardware checks out and the machine is still slower than it should be, the question moves from health to tuning. Optimizing server performance goes over that side.

Read the controller rather than the operating system

On a machine with a hardware array, the operating system sees one healthy device regardless of what the disks underneath are doing.

which megacli perccli storcli ssacli 2>/dev/null
storcli /c0 show 2>/dev/null | head -30
perccli /c0/vall show 2>/dev/null

The array state, the individual drive states and the battery on the cache module are all reported here and nowhere else. A degraded array is invisible from the usual system tools, which is exactly why it goes unnoticed until the second failure.

The cache battery is worth its own attention. When it fails, the controller disables write caching to protect data, and the machine becomes dramatically slower overnight with no other explanation. That is a common and confusing incident with a one line diagnosis.

Watch the counters that move before something fails

Most hardware failures are preceded by numbers that climb, and the value is in the change rather than in the absolute figure.

smartctl -A /dev/sda | awk '$1 ~ /^(5|187|188|197|198)$/ {print $2, $10}'
ip -s link show | awk '/^[0-9]+:/{i=$2} /RX:|TX:/{getline; print i, $3, $4}'

Reallocated sectors, pending sectors and uncorrectable errors on a disk mean the drive is consuming its spare capacity. Errors and drops on a network interface mean a cable, a port or a duplex mismatch.

Record these weekly. A count that has been the same number for a year is history and can be ignored. The same count rising over a fortnight is a component telling you it is going, and replacing it during a maintenance window is a different event from replacing it during an outage.

Confirm the recovery paths before you need them

Monitoring tells you something broke. What matters next is whether the things you assume will work actually do.

Three assumptions are worth testing deliberately, at a time you choose. That the machine boots unattended, without waiting at a prompt for a filesystem or a missing device. That the out of band console is reachable and the credentials work. And that a spare drive is in the building.

systemctl is-enabled --quiet nginx mysql && echo "servisler acilista aktif"
findmnt --verify
systemd-analyze verify default.target 2>&1 | head

Each of these has failed for somebody at the worst moment, and each takes a few minutes to verify while everything is working. Using IPMI and out of band management deals with the second.