A site becomes unreachable under load and the first conclusion is an attack. Most of the time it is not, and the distinction decides whether you spend money or enable caching.
Look at the log before concluding anything
tail -1000 ~/logs/example.com | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
tail -1000 ~/logs/example.com | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
The first command shows which addresses. The second shows which paths. Between them the answer is usually obvious in thirty seconds.
Three patterns
Real visitors. Requests spread across many pages, referrers that make sense, addresses distributed like your audience, and the traffic appears in analytics. Something was shared somewhere.
The response is caching. A site serving cached pages handles many times the traffic on the same hosting. There is more in handling a traffic spike.
A crawler. A small number of addresses fetching page after page in order, usually identifying itself in the user agent. Not malicious, occasionally expensive.
Rate limiting handles it, and caching removes most of the cost. There is more in stopping bots and scrapers.
An actual attack. An enormous number of requests, frequently to one expensive path; a search page, a login, anything that cannot be cached. Addresses with no relationship to your audience. And nothing in analytics, because the requests never render a page.
Why filtering must happen upstream
This is the part that decides what can be done.
Traffic that reaches your server has already consumed the bandwidth you are paying for. Blocking it at the server stops it consuming processing and does nothing about the connection being saturated.
So genuine mitigation happens before your server: at a content delivery network, at your host's network, or at a filtering service. That is why the answer to a real attack is rarely something you configure yourself.
Which is also why the diagnosis matters. An uncached site meeting ordinary load is fixed by you for nothing; an attack is handled by someone upstream.
What to do while it is happening
Tell your host. They can see the network and you cannot. They may already be aware, and on shared hosting they will be acting whether you ask or not.
Turn on the most aggressive caching you have, including for pages you normally keep dynamic. A page served from cache costs a fraction of one that runs.
Protect the expensive paths. If the requests target search or login, rate limit those specifically or disable them temporarily. That is a small loss compared with the site being down.
Do not chase addresses. Blocking them one at a time is unwinnable when the sources number in the thousands, and the effort is better spent on the two items above.
What not to do
Do not block whole countries as a reflex. It blocks customers and travellers, hostile traffic moves, and the rule is usually left in place for years afterwards.
Do not raise resource limits to absorb it. That converts an outage into a larger bill and does not stop anything.
The application-layer version
The harder case is a modest volume of requests that each cost a great deal. A few hundred a second against a search query with no index.
The volume looks unremarkable, so network-level filtering sees nothing wrong, and the site is down.
The defence is making those paths cheap or unreachable: cache what you can, index the queries, and rate limit what remains. Telling whether a slow site is the server or the site sets out finding which path is expensive.
Afterwards
Whatever it was, the thing that made it damaging is worth fixing.
Sites that survive both attacks and popularity have the same property: most requests never reach the application. Caching is the measure that does the most against both, and it is worth arranging before the next occasion rather than during it. Understanding caching layers picks it up from there.
And check whether the site was reachable by address as well as by name. A CDN protects nothing if the origin server answers directly.
Tell the flood apart from the popularity
A surge in traffic is not automatically an attack, and treating a successful campaign as one is an expensive mistake.
awk '{print $1}' ~/logs/example.com | sort | uniq -c | sort -rn | head -10
awk '{print $7}' ~/logs/example.com | sort | uniq -c | sort -rn | head -10
awk -F'"' '{print $6}' ~/logs/example.com | sort | uniq -c | sort -rn | head -5
Legitimate traffic is spread across many addresses, requests many different pages, and carries a variety of browser identifiers. It also arrives from somewhere: a referrer, a campaign parameter, a search engine.
An attack usually concentrates on one path, repeats a small set of identifiers, and has no referrer at all. Those three readings distinguish the two in under a minute, which is the decision that determines everything you do next.
Keep serving something rather than nothing
While filtering is being arranged upstream, the goal is to keep the machine responsive rather than to serve everybody.
ss -tn state established | wc -l ss -tn state syn-recv | wc -l uptime
A large count in the half open state means connections are being started and not completed, which is a different attack from one making complete requests.
Serving a small static page instead of the application removes the database and the application from the path entirely, and a machine that would collapse under dynamic requests will serve a static file at very high rates. That keeps the site technically up and buys the time needed for the real fix.
Write down what happened while it is fresh
The value of an incident is what it tells you, and that is lost within a week if nobody records it.
date; cp ~/logs/example.com /root/incident-$(date +%F).log
awk '{print $1}' /root/incident-*.log | sort | uniq -c | sort -rn | head -20 > /root/incident-sources.txt
Copy the logs before they rotate, since the evidence disappears on a schedule regardless of whether anyone has looked at it.
Record when it started, what the traffic looked like, what you changed, and what actually helped. The next occurrence is handled in minutes rather than hours if that note exists, and by somebody who was not there the first time. Setting up uptime monitoring covers noticing it sooner.