Ahosting Logo
Knowledge Base

How to Plan and Run Server Maintenance Windows

Four things to write down before the window opensDecided in advance, not duringWhat you are changingspecifically; update packages isnot a planHow to undo itthe rollback, tested if possibleHow you will know it workedthe check, decided before youare tiredWhen you stopthe time at which you roll backrather than continueWho is toldand in their time zone, not theserver oneReboot on purposeso an unplanned one later is notthe first testThe hardest decision in a maintenance window is stopping, which is why the stop time is decided beforehand rather thanin the moment.

Maintenance is not optional, kernels, control panels and databases all need updates, and every one carries a chance of something not coming back. A maintenance window is how you make that chance land at a time you chose.

The alternative is not "no maintenance". It is maintenance that happens unplanned, during business hours, after something breaks.

Pick the window from your own traffic

Not from a convention. Look at your analytics and find the genuine trough.

For a business serving one country that is usually the small hours. For an international audience there may be no quiet hour, and the right answer is the least-bad one rather than a quiet one.

Avoid the obvious traps: the end of the month for anything financial, the run-up to a sales period for a shop, and the day before a public holiday when nobody is available if it goes wrong.

Tell people beforehand

A week for anything with expected downtime, and a reminder on the day.

Say what is affected and what is not. "The website will be unavailable for up to 30 minutes; email will continue working" is a message that prevents support contact. "Scheduled maintenance" alone does not.

As a reseller, tell clients rather than assuming they will not notice. A client who was told about a ten-minute outage is fine; the same client discovering it is a ticket and a lost afternoon of goodwill. Reseller support workflow deals with keeping that consistent.

Write the plan down first

Four things, in a file, before you start.

What you are changing, specifically. "Update packages" is not a plan; a list of what will be updated is.

How to undo each step. If a step has no undo, that is worth knowing before you take it rather than after.

How you will know it worked; the specific checks, not "see if the site loads".

When you will stop and roll back. A time, decided in advance, at which an unresolved problem becomes a rollback rather than another hour of investigation.

That last one matters most. At two in the morning, three hours into a problem, nobody makes that decision well without having made it earlier.

Back up immediately before, not last night

A backup from the previous night loses everything since. Take one at the start of the window.

And confirm you can reach it from somewhere other than the machine you are working on. A backup accessible only through the server you just broke is not available at the moment it is needed. Backups and disaster recovery walks through where it should live.

Confirm your way back in

Before touching anything: log in to the out-of-band console and confirm it works.

Kernel updates, firewall changes and network configuration can all leave a machine running and unreachable over SSH. IPMI sits beneath the operating system and is what recovers that, but only if the password works and the console opens, which is not the time to find out. Using IPMI deals with testing it.

Reboot on purpose

The step people skip because the server is working.

Many updates only take effect at a restart, and a server that has been up for a year has accumulated changes nobody has tested against a boot. Services that will not start, a fstab entry that is wrong, a firewall rule that was never persisted: all invisible until a reboot happens.

Reboot during your window, while you are watching, rather than discovering it during an unplanned restart six months later.

One change at a time

Tempting to batch everything into one window and hard to diagnose when the result is broken.

Apply a change, verify, then the next. When something breaks you know which one did it, and the rollback is one step rather than all of them.

If the window is too short for that, the window is too short, not a reason to batch. There is more on one change that deserves its own window entirely in EasyApache and PHP versions.

Verify properly afterwards

Load a real page on several sites, not one. Log in to an admin area. Submit a form. Send and receive mail.

Then check that services actually started rather than that the update reported success, those are different things, and a service that failed to start looks fine until someone visits the site it serves.

Watch the error logs for the rest of the day. A missing extension produces a fatal error only on the page that needs it, which may not be the homepage. Troubleshooting common issues walks through what to do when something did not come back.

Record what you did

What changed, when, and anything that surprised you.

Two reasons. A problem appearing three days later is far easier to explain with a record of what changed. And the next window is quicker when you are not rediscovering what you learned in this one.

A dated text file is enough. The formality is not the point; the record is.

A monthly rhythm beats an annual scramble

Small, regular maintenance is safer than a large annual one, because each window changes less and the rollback is smaller.

Security updates as they appear, larger updates monthly, a reboot at least quarterly so boot problems surface while you are watching.

Servers that go a year without maintenance are not stable. They are accumulating a large, untested change that will all be applied at once, eventually, under pressure.

The technical window is half of it; what clients are told decides how the work is remembered. How to Tell Clients About Maintenance and Outages explains that half.