Ahosting Logo
Knowledge Base

How to Diagnose a Server That Will Not Boot

When SSH stops answering there is no information, unless you use the consoleIs the machine running at allthe provider panel sayspowered on or notOpen the consolethis is the part peopleforget existsRead what it stopped ona filesystem check, afailed mount, a serviceloopBoot to rescue if neededmount the disk and fixthe file that broke itThe two usual causes are a full disk and an edit to a boot-critical file, and both are visible on the console withinseconds.

SSH does not answer. That is not a diagnosis; it is the same symptom for a machine that is switched off, one that booted without networking, and one waiting at a prompt nobody can see.

Open the console first

Every provider offers console access: a view of the machine's screen, delivered independently of its network. On a dedicated server this is the management controller. There is more in using IPMI and out-of-band management.

This is the single most useful thing available during an outage and the one people forget exists, because they have never needed it.

Find it before you need it. Hunting for it while a server is down, possibly needing a password reset to reach the provider's panel, is a bad start.

What the console tells you

A login prompt. The machine booted. The problem is networking, the firewall, or the SSH service, and you can log in at the console and look.

Boot messages that stopped. The last line before it stopped names the cause. This is usually the whole answer, written out in plain language, and it is why the console matters more than any guess.

A prompt about the filesystem. The disk needs checking, and the machine will wait there indefinitely. Run the check it suggests.

Nothing at all. The machine is off, or not getting as far as producing output. Power it on from the provider's panel and watch.

The three usual causes

A full disk. Services cannot write and fail to start. The boot may complete with a login prompt and nothing working.

df -h
df -i

Check both: running out of inodes produces the same failures with space still free, and the error says the disk is full either way. Understanding inodes explains it.

An unclean shutdown. The filesystem needs checking, which is why the boot is waiting. Answer the prompt or run the check manually.

A file edited just before the reboot. The most common of the three by a distance. A configuration change that seemed fine while the service was running prevents it from starting again.

The strong clue is timing: if the last thing anyone did was edit something, that is where to look first, whatever else the messages say.

Rescue mode

When the machine will not boot far enough to log in, rescue mode boots a separate small system with your disk attached as storage.

You can then mount your filesystem and fix the file that is preventing the boot:

mkdir /mnt/root
mount /dev/vda1 /mnt/root
nano /mnt/root/etc/fstab

This is how you undo the edit, clear space, or read the logs from the failed boot, where server logs live explains which file, and journalctl can be pointed at the mounted system's journal.

An fstab entry for a disk that no longer exists is a classic cause of a machine that hangs during boot, and it is only fixable this way.

When it boots but SSH does not answer

Log in at the console and work outwards:

ip addr
systemctl status sshd
ss -tlnp | grep :22
iptables -L -n | head

The order matters. No address means networking; SSH not running means the service; running but not listening means configuration; listening but unreachable means the firewall.

A firewall rule applied without being made permanent, or one made permanent with an error in it, is the usual answer here, and it is why firewall changes should be tested before they are saved.

Afterwards

Reboot deliberately once the machine is working, at a quiet moment, and confirm everything comes back on its own.

A service that was started by hand and never enabled works perfectly until the next reboot, which is frequently how a fault like this began. There is more on making that impossible in running an application as a systemd service.

And write down what happened. The next occurrence, months later, otherwise starts from the beginning. Common VPS issues deals with the rest of the list.

Establish whether the machine is running at all

Before opening a console, two checks from outside distinguish a powered-off machine from a running one that is unreachable.

ping -c 3 203.0.113.10
nc -zv -w 3 203.0.113.10 22 2>&1 | tail -1

A machine answering ping with no ports open has booted far enough for networking and is failing later: frequently a filesystem check or a service hanging.

No response at all means it is off, not routed, or has not reached the point of configuring the network. That distinction determines whether the console will show a login prompt or boot messages, and it takes ten seconds.

Boot into an earlier state

When the machine stopped booting after an update, the previous kernel is usually still installed and selectable.

At the boot menu, choose the earlier entry. If the machine boots, the new kernel or its modules are the cause, which narrows the problem to something specific and gives you a working system to fix it from.

ls /boot/vmlinuz-*
grep -c '^menuentry' /boot/grub2/grub.cfg 2>/dev/null || grep -c '^menuentry' /boot/grub/grub.cfg

Reaching the menu requires the console, and on a virtual machine the window is brief, which is why having the console open before rebooting a machine you have just updated is worth the small effort. Setting up automatic security updates goes into scheduling those reboots deliberately.

Boot without the thing that is failing

A machine that hangs waiting for a service or a filesystem can frequently be started past it.

Editing the boot entry to add single or systemd.unit=rescue.target starts a minimal system with local access and without the services that hang. From there the offending unit can be disabled and the machine rebooted normally.

For a filesystem entry that no longer resolves, adding nofail to it in /etc/fstab means a future absence delays the boot rather than stopping it, which is worth doing on any mount that is not essential, precisely because it converts this class of outage into a warning.

Write down what recovered it

Boot failures recur, frequently months later and frequently to a different person.

Record what the console showed, which route recovered it, and what was changed. Three lines, kept with the server's documentation rather than in a personal note.

The specific thing worth recording is where the provider's console is and how to reach it, because that is the step that costs the most time when nobody has done it before. There is more on the equivalent on your own hardware in using IPMI and out-of-band management.