Ahosting Logo

Dedicated Server

Troubleshooting Common Dedicated Server Issues

The first split, and the group a VPS never hasDoes SSH connect?YesThe machine is runningcheck disk first, then services, then what changedNoUse the management consoleand now hardware is on the list: array, memory,power, bootOn a dedicated server the second branch includes failures a VPS simply hides from you, which is why the console is thefirst tool rather than the last.

Dedicated server problems split into two groups that are diagnosed completely differently: software failures, which look like any other Linux server, and hardware failures, which a VPS simply does not have. Working out which you are in takes about a minute.

Can you reach it at all

ping 203.0.113.45
ssh [email protected]

If SSH works, the machine is running and the fault is a service on it.

If nothing responds, go to the out-of-band management interface. That is what it exists for, console access independent of the operating system tells you immediately whether the machine is powered on, whether it booted, and what the console says.

A server that is powered on with an error on the console is a very different problem from one that is powered off, and you cannot distinguish them from outside.

Software: check disk first

df -h

A full filesystem stops the web server, the database and mail simultaneously, and none of the resulting errors mention disk. It is the cheapest check and it explains the most confusing failures.

Then memory. A process that vanished without an error in its own log was killed by the kernel, not failed, check the system log for out-of-memory entries.

Then whether the service is running, and its own log's last twenty lines.

Hardware: what a dedicated server can do that a VPS cannot

A degraded RAID array. The server works normally with a failed disk (that is the point) so nothing tells you unless monitoring is set up. Check array status explicitly when investigating anything unexplained.

Failing memory. Produces random crashes and corruption that look like software bugs and are not. If the system log shows memory errors, believe them.

Disk errors. Read errors in the system log precede failure, often by days. That warning window is when to act.

Thermal or power problems. Visible in the management interface's sensor readings. A server rebooting at intervals with no software cause is usually one of these.

Intermittent problems are usually hardware

Software fails consistently. A configuration error breaks the same way every time.

Random reboots, occasional corruption, and crashes with no pattern in the application logs point at memory, disk or power. Chasing those in software wastes days, and the sensor readings and system log answer it directly.

Report it properly

A hardware issue needs the datacentre. What turns a long exchange into one reply:

The exact log lines showing the error. The array status if a disk is suspected. The sensor readings if it is thermal. When it started and whether it is reproducible. And whether the machine is currently reachable, since that decides whether anything can be done remotely.

Before a disk is replaced

Confirm which disk. Pulling the wrong one from a degraded array destroys it, and that is unrecoverable.

Check backups are current first. A rebuild is the most fragile the array will ever be, and a second failure during it loses everything. Setting up RAID and storage goes into what a rebuild involves.

The check that prevents most of this

Monitoring that alerts on array state, disk errors and sensor thresholds. All three failures above are silent by default, and all three give warning if something is watching.

A degraded array found by monitoring is a scheduled disk swap. The same array found when the second disk fails is a restore from backup.

Read the hardware event log first

The management controller keeps its own record, entirely separate from the operating system's, and it is where hardware faults are written.

ipmitool sel list | tail -30
ipmitool sel elist | grep -i -E 'error|fail|critical'

Memory errors, power supply failures, temperature events and disk removals all appear here, frequently long before anything shows up in the system's own logs, and sometimes when nothing shows up there at all.

This is the log to check when the operating system reports nothing and the machine is misbehaving. Clear it after a resolved incident so the next reading is not obscured by history. There is more on reaching it when the machine is unresponsive in using IPMI and out-of-band management.

Isolate a suspect memory module

When errors point at memory, the count usually names which module, and testing the theory does not require a replacement part.

edac-util -v
dmidecode -t memory | grep -E 'Locator|Size' | head -20

Match the reported location against the physical slot, then either remove that module or swap it with another and see whether the errors follow the module or stay with the slot.

Errors that follow the module mean a faulty module. Errors that stay with the slot mean a board or seating problem, which is a different conversation with the provider, and one that a swap resolves before a replacement is shipped for nothing.

Distinguish a network problem from a server problem

A machine that is intermittently unreachable is frequently blamed for a network fault.

mtr -r -c 100 8.8.8.8
ping -c 100 -i 0.2 GATEWAY-IP | tail -3
ethtool eth0 | grep -E 'Speed|Duplex|Link'

Loss that begins at the first hop is local. The cable, the card, or the switch port. Loss that begins further out is somebody else's network and no amount of work on the server will change it.

A link reporting a lower speed or half duplex than expected is a negotiation fault, and it produces exactly the symptom of a server that is fine and slow, worth checking before anything else when throughput is the complaint.

Write down what you tried

Intermittent hardware faults are diagnosed over days, frequently by more than one person, and the useful information is the pattern instead of any single observation.

Keep a note per incident: what happened, when, what the event log said, what was changed. After three incidents the pattern is usually obvious. A particular time of day, a particular load, a particular disk.

Without the record, each occurrence is investigated from the beginning, and the provider receives three unrelated reports instead of one convincing one. Monitoring server health deals with the readings worth capturing while it is happening.