robots.txt sits at the root of a site and tells crawlers what they may fetch. It is small, it is widely misunderstood, and the misunderstanding usually makes things worse rather than better.
Crawling is not indexing
A Disallow rule says "do not fetch this". It does not say "do not list this".
If the blocked page is linked from anywhere, search engines can still include it in results, typically as a bare address with no description, because they were forbidden from reading it.
So blocking a page you want hidden produces the opposite of the intended effect: it appears in results, looking broken.
To keep a page out of results
The page must be crawlable and must say so itself:
<meta name="robots" content="noindex">
Or as a header, which works for files that are not HTML:
X-Robots-Tag: noindex
The crawler has to fetch the page to see either. Which means a page blocked in robots.txt can never be removed this way: the instruction is there and nobody is allowed to read it.
The sequence for removing an already-indexed page is therefore: allow crawling, add noindex, wait for it to be re-crawled, and only then block it in robots.txt if you still want to.
A reasonable file
User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ Disallow: /*?s= Disallow: /*?orderby= Sitemap: https://example.com/sitemap.xml
Personal pages, and the query patterns that generate endless variations of the same listing.
The Sitemap line is the most useful thing in the file and the most often missing. It is an absolute URL, and it applies regardless of the user-agent blocks.
What it is genuinely for
Crawl effort on large sites. A crawler has a finite budget for your site. Spending it on thousands of filtered listing combinations means less of it on the pages you care about.
Expensive endpoints. An internal search page crawled repeatedly runs a database query every time. Blocking it removes real load, and this is a hosting benefit as much as an SEO one.
Faceted navigation. Filter combinations multiply into effectively infinite addresses. This is the main reason large stores need the file at all. There is more on the store side in improving WooCommerce search and filters.
Do not block assets
Blocking CSS and JavaScript directories was once common advice and is now actively harmful. Search engines render pages, and a page rendered without its stylesheet is judged as what they can see, frequently as not mobile-friendly.
Let them fetch everything the page needs to display.
It is a public file
Anyone can read example.com/robots.txt, and automated scanners do so first.
Listing administrative paths there hands over a list of what you consider sensitive. It is not obscurity; it is a directory.
Anything that genuinely must not be reached needs authentication or an access restriction. There is more on the real mechanism in password protecting a directory.
Rules that get people
Disallow: / blocks the entire site. It appears on staging sites and is occasionally copied to production during a launch, which removes the site from search results within days. If organic traffic disappeared after a redesign, check this file first.
Allow can carve exceptions out of a broader block, and the most specific matching rule wins:
Disallow: /files/ Allow: /files/public/
Crawl-delay is honoured by some crawlers and ignored by the largest ones, so it is not a way to control aggressive crawling. Stopping bots and scrapers deals with what actually works.
Check it
curl https://example.com/robots.txt
Confirm it is served at the root, over the canonical hostname, and that it returns 200 instead of a 404 page containing HTML. A robots file that is really an error page is read as an unparseable file, and behaviour then varies by crawler.
Each subdomain needs its own file. The one at the main domain does not apply to shop.example.com.
The file only covers its own host
A robots file applies to the scheme, host and port it was fetched from; nothing else.
So example.com/robots.txt does not govern shop.example.com, and the secure and plain versions are technically separate. Each subdomain needs its own file, served from its own root.
This is how staging sites end up indexed: the main site's file disallows everything sensible, the staging subdomain has no file at all, and crawlers treat that as permission. There is more on why staging belongs on one, and what else it needs in choosing between a subdomain and a subfolder.
Rule precedence is not top to bottom
Unlike most configuration files, the order of rules does not decide the outcome. The most specific matching rule wins, regardless of position.
Disallow: /files/ Allow: /files/public/
The allow rule wins for anything under the public path because it matches more characters, and it would still win if written first.
Two wildcards are worth knowing: * matches any sequence, and $ anchors the end of the address.
Disallow: /*.pdf$ Disallow: /*?print=
The second is the useful pattern for query strings that produce duplicate versions of a page.
User-agent groups do not combine
A crawler obeys exactly one group. The most specific one that names it, and ignores the rest entirely.
User-agent: * Disallow: /cart/ User-agent: Googlebot Disallow: /internal/
Googlebot here is not disallowed from the cart, because it follows its own group and the general one does not apply to it.
That surprises people regularly. If a rule should apply to everybody, it must be repeated in every group that exists, which is a good argument for keeping the file to a single group unless there is a specific reason not to.
Check it as a crawler sees it
curl -sI https://example.com/robots.txt curl -s https://example.com/robots.txt | head -20
Three things to confirm. It returns 200 instead of a 404 page containing HTML. A robots file that is really an error page is treated inconsistently. It is served as plain text rather than as HTML. And it is reachable over the canonical hostname, since a redirect to another host is followed by some crawlers and not others.
Then check the same for every subdomain that exists, which is the part usually skipped. There is more on enumerating them in SSL for subdomains and addon domains.