A site is slower than it was, and analytics shows the same number of visitors. The traffic that is not in analytics is doing the work.
On most sites automated requests outnumber human ones. The question is not how to stop them, some of them you need, but which are costing you.
Find out what is actually hitting you
awk '{print $1}' ~/logs/example.com | sort | uniq -c | sort -rn | head -20
grep -oiE 'bot|crawler|spider|scrap' ~/logs/example.com | sort | uniq -c | sort -rn | head
One address responsible for a large share of all requests is the pattern you are looking for. A genuine visitor generates a handful of page requests; a scraper generates thousands.
Also look at what they request. A crawler fetches pages. A scanner fetches wp-login.php, /.env and administrative paths that do not exist on your site. Viewing website statistics sets out reading the log.
Three categories
Keep. Search engine crawlers, and the services that fetch a page to build a link preview when someone shares it. Blocking these makes you invisible in search and makes your shared links look broken.
Tolerate or limit. Analytics services, monitoring, feed readers. Not hostile, occasionally expensive.
Stop. Scrapers copying the entire site, vulnerability scanners, and automated login attempts.
Why user-agent blocking barely works
The user agent is a string the client sends. Anything can claim to be anything.
Blocking by user agent stops the well-behaved bots that identified themselves honestly, and does nothing at all to the ones you wanted to stop, which claim to be Chrome.
It is worth doing for the specific known services you want to exclude. It is not a defence.
What actually works
Caching. The most effective measure, and the least discussed in this context. A cached page served to a crawler costs almost nothing; no PHP, no database. Crawling stops being a resource problem entirely.
Most sites with a bot problem have a caching problem. Understanding caching layers walks through it.
Rate limiting by address. Allow a reasonable number of requests per minute and slow down anything beyond it. This is agnostic about who the client claims to be, which is precisely why it works.
Blocking before the application. A request refused at the web server costs a fraction of one that starts PHP. Blocking inside the application means paying most of the cost before deciding not to serve it. There is more on the panel's version in IP Blocker in cPanel.
Protect the expensive paths first
Not all requests cost the same. A cached article is nearly free; an internal search query runs a database query that cannot be cached usefully.
Scrapers hit search and filter pages hard, because that is how they enumerate a site. Blocking crawling of those specific paths often removes most of the load while leaving your actual content fully crawlable. Writing a robots.txt walks through doing it for the bots that obey, and rate limiting covers the ones that do not.
Login pages
Automated login attempts are constant on every site and each one starts PHP, which makes them expensive as well as dangerous.
Rate limiting the login path specifically is worth more than any general measure, because that is where the volume concentrates. Improving WordPress security goes over the application side, and note that panel-level protection does not see these requests at all. They are ordinary web traffic to a PHP file.
Blocking whole countries and networks
Sometimes proposed and rarely right. It blocks people as well as bots, including customers travelling and anyone using a VPN, and hostile traffic simply moves.
Blocking a specific hosting network that is sending nothing but scrapers is more defensible, real visitors rarely browse from a data centre. Even then, review it periodically instead of leaving it permanently.
Judge it by cost, not by principle
The goal is not zero bots. It is that automated traffic stops affecting real visitors.
If caching means a crawler costs you nothing, there is no reason to fight it. Spend the effort where requests are expensive and the traffic gives nothing back. Handling a traffic spike goes into telling the two apart when load rises suddenly.
Serve a cheap answer instead of a full page
Blocking is not the only option, and it is frequently not the best one.
A request refused with a small response costs almost nothing, and a slow response costs the client far more than it costs you. Both are better than serving a full page to something that gives nothing back.
RewriteCond %{QUERY_STRING} (^|&)orderby= [NC]
RewriteCond %{HTTP_USER_AGENT} !(googlebot|bingbot) [NC]
RewriteRule ^ - [R=429,L]
A 429 tells a well-behaved client to slow down and costs you a header in place of a database query. On the filtered listing pages that scrapers walk, that alone removes most of the load.
Verify a crawler that claims to be one
Anything can put a search engine's name in its user agent, and aggressive scrapers routinely do: precisely because sites exempt those names from rate limits.
The check is a reverse lookup followed by a forward one:
host 66.249.66.1 host crawl-66-249-66-1.googlebot.com
A genuine crawler's address resolves to the operator's domain, and that name resolves back to the same address. A pretender fails one or both.
Doing this before granting an exemption is what stops the exemption becoming the way in, and it is worth automating if you maintain such a list at all.
Read the log for the pattern, not the volume
awk '{print $1}' ~/logs/example.com | sort | uniq -c | sort -rn | head -10
awk '$9>=400 {print $1, $9, $7}' ~/logs/example.com | sort | uniq -c | sort -rn | head -10
awk '{print $1}' ~/logs/example.com | sort -u | wc -l
The second command is the useful one and the least used: sources generating errors are usually probing rather than browsing, and a single address producing hundreds of 404s for paths that never existed is not a visitor.
The third gives the number of distinct sources. A large request count from few addresses is a crawler or a scraper; a similar count from thousands of addresses is a different problem. Recognising and handling a DDoS attack goes into that case.
Blocking that outlives its reason
Rules added during an incident are rarely removed, and they accumulate.
The cost appears later and is hard to attribute: a customer who cannot reach the site from their network, a payment provider's callback refused, a monitoring service treated as an attacker.
Date every rule you add, in a comment, with why. Review them once a year and remove anything whose reason no longer applies. A block list that only grows eventually blocks somebody who matters, and nobody will connect it to a rule added two years earlier.