Ahosting Logo
Knowledge Base

How to Test a New Server Before Putting It into Service

Components that fail tend to fail earlyMemory testfor hours, because badmemory corrupts quietlyDisk surface and SMARTbefore any data is onitSustained loadso thermal and powerproblems appear nowYour actual applicationa synthetic benchmarkproves the hardware,not the fitA machine that has run under load for a day without complaint is far more trustworthy than one that booted and lookedfine.

A new machine is the most likely moment for a component to fail and the least costly moment to discover it. A day of deliberate load before anything moves onto it is the cheapest insurance available.

Memory first

A faulty memory module does not fail cleanly. It corrupts data occasionally, which presents months later as unexplained crashes, database corruption and application errors that nobody can reproduce.

It is the hardest fault to diagnose after the fact and the easiest to find now.

memtester 4G 3 # while the system is running
# or a full pass from a boot image, which tests all of it

A full pass on a large machine takes hours. Start it and leave it. On server hardware, also confirm error correction is active and check the counters afterwards: monitoring server health deals with reading them, and a machine that already reports correctable errors on day one should be replaced rather than accepted.

Storage

fio --name=rw --rw=randrw --bs=4k --size=4G --numjobs=4 --runtime=300 --group_reporting
smartctl -a /dev/sda | grep -iE 'reallocated|pending|health'

Two separate questions: does the storage perform as expected, and does each drive report itself healthy.

A new drive with a non-zero reallocated count is not a new drive. That is worth raising immediately rather than after it fails.

Processor and cooling together

stress-ng --cpu 0 --timeout 3600s --metrics-brief

An hour at full load tells you two things. Whether the machine is stable under sustained work, and whether it stays cool enough to run at its rated speed.

Watch the temperatures and the clock speed while it runs:

ipmitool sdr type temperature
watch -n5 'grep MHz /proc/cpuinfo | head'

A processor reducing its own speed to protect itself means a cooling problem, and that fault is otherwise discovered months later as a server that is inexplicably slower than it used to be, which is almost always blamed on software.

Network

iperf3 -c speedtest-host -t 60
mtr -r -c 100 8.8.8.8

Confirm the connection performs as specified rather than as sold, and that the route is clean. Packet loss on a new machine is a provider conversation, and it is much easier to have before you depend on the machine.

Understanding bandwidth, port speed and network limits covers what the figures mean.

Fail a drive on purpose

The test most worth doing and least often done.

With the array healthy and no data at risk, mark a drive failed and watch the rebuild. Confirm it starts, completes, and that the system remained usable throughout.

What you are testing is not the disk but your own understanding: whether you know the commands, whether the monitoring alerted, and how long a rebuild takes. All three are things you will want to know during a real failure and cannot look up calmly then.

Setting up RAID and storage management deals with the arrangement.

Confirm the recovery paths

Two more, before the machine matters.

The console. Log into the management interface and confirm you can see the screen and power the machine off and on. That is the route you will need when it stops responding. For that, see using IPMI and out-of-band management.

A reboot. Restart deliberately and confirm everything comes back on its own. A service started by hand and never enabled works perfectly until the first unplanned restart. Running an application as a systemd service deals with making that impossible.

Record the results

Temperatures under load, storage throughput, network figures, memory error counts.

These are the baseline. Six months later, when the machine feels slower, the only way to know whether it actually is comes from having measured it when it was new. For making those measurements comparable, see benchmarking a server meaningfully.

Write the results somewhere permanent

The tests only pay off if the figures survive, and a terminal scrollback is not where they survive.

{
 echo "=== $(hostname) commissioned $(date -u +%F)"
 lscpu | grep -E 'Model name|^CPU\(s\)'
 free -h | head -2
 lsblk -f
 ipmitool sdr type temperature 2>/dev/null | head
} > ~/commissioning.txt

Keep it off the machine as well as on it. When the server is unreachable a year later, the record of what it contained is what you need to order a replacement or brief a technician.

It is also the baseline every later comparison depends on. Benchmarking a server meaningfully goes into making those figures comparable.

Test it as it will actually run

Synthetic tests exercise components. They do not exercise the combination, which is where problems appear.

Install the real stack and put representative load through it before the machine carries anything. A web server, a database and the application, with a load generator producing traffic that resembles the site's.

What that finds and the component tests do not: a configuration that works until several things run at once, memory allocation that is fine individually and exceeds the machine together, and a service that fails under sustained connections rather than under a burst.

The pool sizing is the specific setting worth exercising here, because its failure mode is queuing instead of an error. There is more on the number in web server capacity.

Verify the provider gave you what was ordered

Worth checking rather than assuming, and it takes two minutes.

lscpu | grep -E 'Model name|^CPU\(s\)|Thread'
free -g | head -2
lsblk -d -o NAME,SIZE,ROTA,MODEL

Processor model, core count, memory quantity and disk model against what the order said. Discrepancies happen. A different processor generation, fewer cores, spinning disks where solid-state was specified, and they are far easier to raise before the machine is in service.

The ROTA column is the one people miss: a value of 1 means a rotating disk, which is a materially different machine from what most orders assume.

Leave it running before trusting it

A machine that passes every test and has been powered on for an hour has not demonstrated much.

Leave the load tests running overnight. Components that fail early frequently fail after some hours of sustained work rather than in the first minutes, and thermal problems in particular only appear once everything has warmed up.

Check the hardware event log in the morning rather than only the test output. A corrected memory error recorded during the night is the finding, and nothing in the test results mentions it. Troubleshooting common dedicated server issues deals with reading it.