{"id":1229,"date":"2026-09-04T17:27:07","date_gmt":"2026-09-04T17:27:07","guid":{"rendered":"https:\/\/www.ahosting.net\/blog\/?p=1229"},"modified":"2026-09-04T19:50:11","modified_gmt":"2026-09-04T19:50:11","slug":"ai-crawler-traffic-hosting-cost","status":"publish","type":"post","link":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/","title":{"rendered":"AI Crawler Traffic Is Now a Hosting Cost: 65,044 Fetches, Measured"},"content":{"rendered":"\n<script type=\"application\/ld+json\">{\"@context\":\"https:\/\/schema.org\",\"@type\":\"FAQPage\",\"mainEntity\":[{\"@type\":\"Question\",\"name\":\"Do AI crawler visits actually cost me anything on shared hosting in 2026?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"Typically, only where the request reaches PHP. A page served from the server-level cache never starts a dynamic script, so it consumes no entry process and costs the plan nothing measurable. The cost appears on the paths caching does not cover, which is a much smaller and much more specific list than the raw fetch count suggests.\"}},{\"@type\":\"Question\",\"name\":\"AI crawler vs search engine crawler: what is the difference for my server?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"In practice, the load behaves the same and the control does not. A search crawler is building an index and comes back on its own schedule, so it obeys robots.txt and it can be slowed. A user-triggered AI crawler arrives because a person asked a question a moment ago, and several vendors state that this class of fetcher does not treat robots.txt as binding.\"}},{\"@type\":\"Question\",\"name\":\"Can I block AI crawler traffic with robots.txt, and does it work?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"Specifically, it works for the indexing half and not reliably for the other. Perplexity's own documentation says its user-triggered fetcher generally ignores robots.txt rules because a person requested the page. Blocking is therefore a policy statement about training and indexing rather than a load control, and treating it as a load control is where most advice on this topic goes wrong.\"}},{\"@type\":\"Question\",\"name\":\"What is the AHosting robots.txt policy on AI crawler access in 2026?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"Notably, AHosting allows the assistants that cite and link, and blocks the ones that only harvest. The live file explicitly allows GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Applebot-Extended, and disallows CCBot, Bytespider, TikTokBot, anthropic-ai, cohere-ai, meta-externalagent and Amazonbot. It also carries a Content-Signal line reading search=yes, ai-train=no, ai-input=yes.\"}},{\"@type\":\"Question\",\"name\":\"User-triggered fetches vs indexing crawls: which one should I worry about?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"Ultimately, the user-triggered half, because it is both larger and less controllable. Across 28 days on AHosting the three user-triggered agents accounted for 46,876 of 65,044 total fetches. That is the portion that arrives in response to a real question, cannot be scheduled away, and is the only portion with any chance of sending a reader back.\"}},{\"@type\":\"Question\",\"name\":\"How many entry processes does an AI crawler fetch consume on a cached page?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"Fortunately, none. CloudLinux defines an entry process as a concurrent connection to a dynamic script, and a page returned by the server-level cache never invokes one. Where the fetch lands on an uncached path, though, it consumes exactly the same entry process a human visitor would, and it competes for the same ceiling.\"}},{\"@type\":\"Question\",\"name\":\"Which AHosting plan tier gives enough entry processes for heavy AI crawler traffic in 2026?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"Accordingly, the answer depends on your uncached surface, not on the fetch count. The shared tiers allocate 20, 30, 40 and 50 entry processes for Bronze, Silver, WooStart and Gold respectively, verified against the live server configuration. A site whose cache covers its article pages rarely needs a larger tier for crawler load alone; a site with heavy search or faceted URLs often does.\"}},{\"@type\":\"Question\",\"name\":\"How many visitors does AI crawler traffic actually send back to a hosting site?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"Indeed, far fewer than the read volume implies. The same 28-day window that recorded 65,044 AI fetches on AHosting recorded 74 referred visits from ChatGPT and 5 from Gemini. That ratio is the honest starting point for any conversation about whether to allow, throttle or block, and it is a number most sites have never worked out.\"}},{\"@type\":\"Question\",\"name\":\"How do I measure AI crawler load in my own server access logs?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"For example, group raw access-log requests by user agent over a fixed window, then separate the user-triggered agents from the indexing ones before you total anything. The vendors publish the exact agent strings and their IP ranges, so verification is possible rather than guesswork. Measuring first is what stops a policy decision being made on an impression.\"}},{\"@type\":\"Question\",\"name\":\"How does AHosting tell an AI crawler apart from an ordinary bot flood?\",\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"By contrast, a flood concentrates on one endpoint and a crawler spreads across content. Automated abuse hammers a login form, an XML-RPC endpoint or a search parameter thousands of times an hour from unverifiable addresses. Legitimate AI crawler traffic identifies itself, publishes its address ranges, and reads article pages, which is why the two need different responses.\"}}]}<\/script>\n\n\n<div class=\"wp-block-aioseo-table-of-contents\"><ul><li><a class=\"aioseo-toc-item\" href=\"#aioseo-what-ai-crawler-traffic-actually-is-and-why-it-is-not-one-thing\">What AI Crawler Traffic Actually Is (And Why It Is Not One Thing)<\/a><ul><li><a class=\"aioseo-toc-item\" href=\"#aioseo-indexing-crawlers-and-user-triggered-fetchers-are-different-jobs\">Indexing Crawlers and User-Triggered Fetchers Are Different Jobs<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-why-that-distinction-decides-what-you-can-control\">Why That Distinction Decides What You Can Control<\/a><\/li><\/ul><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-28-days-of-ai-crawler-traffic-measured-on-one-hosting-network\">28 Days of AI Crawler Traffic, Measured on One Hosting Network<\/a><ul><li><a class=\"aioseo-toc-item\" href=\"#aioseo-the-ahosting-ai-crawler-load-table\">The AHosting AI Crawler Load Table<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-65044-ai-crawler-fetches-against-298-google-clicks\">65,044 AI Crawler Fetches Against 298 Google Clicks<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-what-came-back-79-assistant-referrals\">What Came Back: 79 Assistant Referrals<\/a><\/li><\/ul><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-what-an-ai-crawler-fetch-actually-costs-your-plan\">What an AI Crawler Fetch Actually Costs Your Plan<\/a><ul><li><a class=\"aioseo-toc-item\" href=\"#aioseo-entry-processes-and-why-a-cached-page-costs-none\">Entry Processes, and Why a Cached Page Costs None<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-where-the-ai-crawler-cost-is-genuinely-concentrated\">Where the AI Crawler Cost Is Genuinely Concentrated<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-ai-crawler-entry-process-headroom-by-plan-tier\">AI Crawler Entry-Process Headroom by Plan Tier<\/a><\/li><\/ul><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-why-robots-txt-does-not-fix-the-biggest-ai-crawler-number\">Why robots.txt Does Not Fix the Biggest AI Crawler Number<\/a><ul><li><a class=\"aioseo-toc-item\" href=\"#aioseo-what-perplexitys-own-documentation-says\">What Perplexity&#039;s Own Documentation Says<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-what-robots-txt-still-does-control\">What robots.txt Still Does Control<\/a><\/li><\/ul><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-ai-crawler-traffic-vs-a-bot-flood-which-problem-do-you-have\">AI Crawler Traffic vs. a Bot Flood: Which Problem Do You Have?<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-what-ahosting-actually-ships-for-ai-crawler-load\">What AHosting Actually Ships for AI Crawler Load<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-a-practical-checklist-for-ai-crawler-traffic\">A Practical Checklist for AI Crawler Traffic<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#faq-ai-crawler-traffic-hosting-cost\">Frequently Asked Questions About AI Crawler Traffic<\/a><ul><li><a class=\"aioseo-toc-item\" href=\"#faq-ai-crawler-cost-2026\">Do AI crawler visits actually cost me anything on shared hosting in 2026?<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#faq-ai-crawler-vs-search-crawler\">AI crawler vs search engine crawler: what is the difference for my server?<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#faq-block-ai-crawlers-robots\">Can I block AI crawler traffic with robots.txt, and does it work?<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#faq-ahosting-ai-crawler-policy-2026\">What is the AHosting robots.txt policy on AI crawler access in 2026?<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#faq-user-triggered-vs-indexing\">User-triggered fetches vs indexing crawls: which one should I worry about?<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#faq-entry-processes-bots\">How many entry processes does an AI crawler fetch consume on a cached page?<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#faq-ahosting-plan-tiers-2026\">Which AHosting plan tier gives enough entry processes for heavy AI crawler traffic in 2026?<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#faq-ai-crawler-referrals\">How many visitors does AI crawler traffic actually send back to a hosting site?<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#faq-measure-ai-crawler-load\">How do I measure AI crawler load in my own server access logs?<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#faq-ahosting-bot-flood-difference\">How does AHosting tell an AI crawler apart from an ordinary bot flood?<\/a><\/li><\/ul><\/li><\/ul><\/div>\n\n\n<div class=\"ah-tldr\">\n  <span class=\"ah-tldr-badge\">TL;DR<\/span>\n  <p>AHosting logged 65,044 AI crawler fetches in 28 days against 298 Google clicks. Most of that load costs nothing, because a cached page consumes no entry process \u2014 and most of it cannot be blocked with robots.txt either, because 72% of it is user-triggered.<\/p>\n<\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Every hosting blog now carries advice about AI crawler traffic, and almost all of it ends at the same instruction: edit your robots.txt. Over 28 days we counted what the AI crawlers actually did to one hosting network \u2014 65,044 fetches \u2014 and then worked out what those fetches cost and which of them that instruction can reach. The answer to the second question is most of the advice does not apply to most of the traffic.<\/p>\n\n\n\n<figure class=\"wp-block-audio\"><audio controls src=\"https:\/\/www.ahosting.net\/blog\/wp-content\/uploads\/2026\/09\/PODCAST-ai-crawler-traffic-hosting-cost.m4a\"><\/audio><figcaption class=\"wp-element-caption\">Listen: 65,044 assistant fetches, 79 visits back, and where the real cost sits. By Matt Chrust, Director of Business Development, AHosting.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Furthermore, the cost is not where the volume is. Most of those 65,044 requests were free, for a reason that has nothing to do with AI and everything to do with how a cached page is served. What follows is the measurement, the mechanism, and the part of the problem that is genuinely worth acting on.<\/p>\n\n\n\n<h2 id=\"aioseo-what-ai-crawler-traffic-actually-is-and-why-it-is-not-one-thing\" class=\"wp-block-heading\">What AI Crawler Traffic Actually Is (And Why It Is Not One Thing)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Notably, the phrase covers two populations that behave alike in a log file and differently in every way that matters. Separating them is the first step, and it is the step most write-ups skip.<\/p>\n\n\n\n<h3 id=\"aioseo-indexing-crawlers-and-user-triggered-fetchers-are-different-jobs\" class=\"wp-block-heading\">Indexing Crawlers and User-Triggered Fetchers Are Different Jobs<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Specifically, an indexing crawler is building a corpus. It arrives on a schedule of its own choosing, works through your site methodically, and stores what it finds for later. <a href=\"https:\/\/platform.openai.com\/docs\/bots\" target=\"_blank\" rel=\"noopener\">OpenAI&#8217;s crawler documentation<\/a> describes exactly this split in its own products: GPTBot gathers content for model training, while OAI-SearchBot exists to surface pages in ChatGPT search results, and a site can allow one and refuse the other.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A user-triggered fetcher does something else entirely. Somebody asked an assistant a question a few seconds ago, the assistant decided your page might answer it, and the fetch happens live while that person waits. <a href=\"https:\/\/docs.perplexity.ai\/guides\/bots\" target=\"_blank\" rel=\"noopener\">Perplexity&#8217;s crawler documentation<\/a> draws the same line: PerplexityBot surfaces and links pages in search results, while Perplexity-User supports a specific action a person just took.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In practice, the log line looks identical. Both are a GET for an article, both come from a declared user agent, and both count once. The difference only appears when you ask what would happen if you tried to stop them.<\/p>\n\n\n\n<h3 id=\"aioseo-why-that-distinction-decides-what-you-can-control\" class=\"wp-block-heading\">Why That Distinction Decides What You Can Control<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Indeed, robots.txt was written for the first population and not the second. <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9309.html\" target=\"_blank\" rel=\"noopener\">RFC 9309<\/a> standardized the Robots Exclusion Protocol for automatic clients working through a site of their own accord, and the whole model assumes a crawler that fetches the file, reads the rules and then decides where to go next.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A fetch made because a person asked for it does not fit that model, and the vendors say so. Perplexity documents that its user-triggered agent generally ignores robots.txt rules, on the stated grounds that a user requested the page. Consequently, the largest category of AI crawler traffic in our own logs is also the category that a robots.txt edit does not reliably touch.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That gap is being worked on rather than merely complained about. The IETF has a working group, <a href=\"https:\/\/datatracker.ietf.org\/wg\/aipref\/about\/\" target=\"_blank\" rel=\"noopener\">AI Preferences<\/a>, building a vocabulary for expressing how content may be used by automated systems, precisely because the existing file cannot carry the distinction. Until that work lands, the honest position is that you have policy control over indexing and very little over live retrieval.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"AI Crawler Traffic: What 65,044 Fetches Actually Cost Us\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube-nocookie.com\/embed\/4PapB9ldyH0?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<h2 id=\"aioseo-28-days-of-ai-crawler-traffic-measured-on-one-hosting-network\" class=\"wp-block-heading\">28 Days of AI Crawler Traffic, Measured on One Hosting Network<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Ultimately, the argument above is only worth making if the numbers behind it are real. These come from AHosting server access logs for the 28 days from 4 August to 31 August 2026, grouped by declared user agent. They are ours, and nothing here is modeled or extrapolated.<\/p>\n\n\n\n<h3 id=\"aioseo-the-ahosting-ai-crawler-load-table\" class=\"wp-block-heading\">The AHosting AI Crawler Load Table<\/h3>\n\n\n\n<figure class=\"wp-block-table ah-ladder\"><table><thead><tr><th>Agent<\/th><th>Fetches, 28 days<\/th><th>Class<\/th><th>Reachable by robots.txt<\/th><\/tr><\/thead><tbody><tr><td>Perplexity-User<\/td><td>38,743<\/td><td>User-triggered<\/td><td>No \u2014 vendor states it generally ignores it<\/td><\/tr><tr><td>ChatGPT-User<\/td><td>7,197<\/td><td>User-triggered<\/td><td>No \u2014 live browsing on a user request<\/td><\/tr><tr><td>Amazonbot<\/td><td>3,743<\/td><td>Indexing<\/td><td>Yes<\/td><\/tr><tr><td>ClaudeBot<\/td><td>3,503<\/td><td>Indexing<\/td><td>Yes<\/td><\/tr><tr><td>PerplexityBot<\/td><td>3,292<\/td><td>Indexing<\/td><td>Yes<\/td><\/tr><tr><td>OAI-SearchBot<\/td><td>2,519<\/td><td>Indexing<\/td><td>Yes<\/td><\/tr><tr><td>GPTBot<\/td><td>2,379<\/td><td>Indexing<\/td><td>Yes<\/td><\/tr><tr><td>Google-Extended<\/td><td>1,020<\/td><td>Indexing<\/td><td>Yes<\/td><\/tr><tr><td>Applebot<\/td><td>988<\/td><td>Indexing<\/td><td>Yes<\/td><\/tr><tr><td>Claude-User<\/td><td>936<\/td><td>User-triggered<\/td><td>No \u2014 live browsing on a user request<\/td><\/tr><tr><td>Bytespider<\/td><td>478<\/td><td>Indexing<\/td><td>Yes<\/td><\/tr><tr><td>meta-externalagent<\/td><td>246<\/td><td>Indexing<\/td><td>Yes<\/td><\/tr><tr><td><strong>Total<\/strong><\/td><td><strong>65,044<\/strong><\/td><td><strong>46,876 user-triggered<\/strong><\/td><td><strong>18,168 reachable<\/strong><\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">The AHosting AI Crawler Load Table \u2014 AHosting.net server access logs, 4\u201331 August 2026 (28 days). Class assignment follows each vendor&#8217;s own published documentation.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Above all, read the last row first. Of 65,044 fetches, 46,876 came from the three agents their own vendors describe as acting on a live user request \u2014 72% of it is user-triggered. The remaining 18,168 are indexing crawls, and those are the ones a robots.txt rule genuinely governs.<\/p>\n\n\n\n<h3 id=\"aioseo-65044-ai-crawler-fetches-against-298-google-clicks\" class=\"wp-block-heading\">65,044 AI Crawler Fetches Against 298 Google Clicks<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">For example, put that total next to the search traffic from the same 28 days. Google Search Console recorded 52,662 impressions and 298 clicks for AHosting.net over the identical window. AI systems therefore read the site roughly 218 times for every visit Google search sent to it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Interestingly, the raw scale is easy to misread as alarming. Spread across 28 days, 65,044 fetches average about 2,323 a day, or roughly 1.6 a minute \u2014 which no server notices as a rate. Volume alone is not the problem, and a post that stopped here would be selling a worry rather than describing one.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Similarly, one agent dominates to a degree worth naming. Perplexity-User alone accounts for 38,743 of the total, close to three-fifths of all AI crawler activity on the network, and it sits in the category that robots.txt does not reliably reach. Any policy that treats this as a single undifferentiated stream will therefore mostly regulate the smaller half.<\/p>\n\n\n\n<h3 id=\"aioseo-what-came-back-79-assistant-referrals\" class=\"wp-block-heading\">What Came Back: 79 Assistant Referrals<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">By contrast, the return traffic over the same window was 74 referred visits from ChatGPT and 5 from Gemini. Seventy-nine visits, against 46,876 user-triggered reads. That works out at one referred reader for roughly every 593 pages an assistant fetched on somebody&#8217;s behalf.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Fortunately, that number is a starting point rather than a verdict. An assistant that answers a question without sending a click still puts the brand in front of the person asking, which is the whole argument of <a href=\"https:\/\/www.ahosting.net\/blog\/optimize-wordpress-for-ai-search\/\">our guide to optimizing a WordPress site for AI search<\/a>. What the ratio does establish is that nobody should assume this traffic pays for itself in visits.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Overall, the useful framing is a trade rather than a leak. You are giving up some server work and getting citations, brand presence and a small number of high-intent readers. Whether that trade is good depends entirely on what the server work costs, which is the next question and the one almost nobody answers.<\/p>\n\n\n\n<h2 id=\"aioseo-what-an-ai-crawler-fetch-actually-costs-your-plan\" class=\"wp-block-heading\">What an AI Crawler Fetch Actually Costs Your Plan<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Therefore the meaningful unit is not the fetch. It is the entry process, and the gap between those two things is where this entire topic gets misjudged.<\/p>\n\n\n\n<h3 id=\"aioseo-entry-processes-and-why-a-cached-page-costs-none\" class=\"wp-block-heading\">Entry Processes, and Why a Cached Page Costs None<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Accordingly, start with the definition rather than the intuition. <a href=\"https:\/\/docs.cloudlinux.com\/cloudlinuxos\/limits\/\" target=\"_blank\" rel=\"noopener\">CloudLinux documents an entry process<\/a> as a limit that usually represents the maximum number of concurrent connections to Apache dynamic scripts, alongside SSH sessions and cron jobs running at the same time. The operative word is dynamic. An entry process is a slot held while PHP is running, not a counter of requests.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On AHosting a cached page costs zero entry processes, because LiteSpeed answers it at the server level and no PHP script is ever started. That is the same mechanism described in <a href=\"https:\/\/www.ahosting.net\/blog\/wordpress-litespeed-hosting-2026\/\">our write-up of server-level caching<\/a>, and it is why the 65,044 figure is not a bill.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Consequently, an AI crawler reading your article pages behaves almost exactly like a visitor reading them from cache: it consumes bandwidth, it does not consume a concurrency slot, and it disappears from the constraint you actually pay for. Where a fetch does reach PHP, it holds one of those slots for as long as the request takes, and at that moment it is indistinguishable from a human.<\/p>\n\n\n\n<h3 id=\"aioseo-where-the-ai-crawler-cost-is-genuinely-concentrated\" class=\"wp-block-heading\">Where the AI Crawler Cost Is Genuinely Concentrated<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">In particular, the cost lives on whatever you expose that caching does not cover. Internal search URLs run a database query per request and are almost never cached. Faceted or parameterized listing pages generate an effectively unlimited set of unique addresses, each one a cache miss by construction. Feeds, calendar archives and admin-ajax endpoints behave the same way.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Moreover, a crawler is far more likely than a human to find those paths, because it follows every link it can see rather than the two or three a reader would. A site with a crawlable faceted archive can hand an AI crawler thousands of distinct uncached URLs from a single category page.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Specifically, this is why the fix is usually a caching and crawl-surface audit rather than a robots.txt edit \u2014 and the two get confused because both are edits to the same kind of file. Where an entry-process ceiling is genuinely being hit, the diagnosis in <a href=\"https:\/\/www.ahosting.net\/blog\/508-resource-limit-reached-wordpress\/\">our explanation of the 508 resource limit error<\/a> applies unchanged, since a crawler-driven ceiling and a traffic-driven one produce the identical symptom.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Notably, an <a href=\"https:\/\/developer.mozilla.org\/en-US\/docs\/Web\/HTTP\/Headers\/Cache-Control\" target=\"_blank\" rel=\"noopener\">appropriate Cache-Control policy<\/a> also determines how often a well-behaved fetcher comes back for something it already has, which quietly reduces the indexing half of the load without refusing anybody anything.<\/p>\n\n\n\n<h3 id=\"aioseo-ai-crawler-entry-process-headroom-by-plan-tier\" class=\"wp-block-heading\">AI Crawler Entry-Process Headroom by Plan Tier<\/h3>\n\n\n\n<figure class=\"wp-block-table ah-ladder\"><table><thead><tr><th>Shared tier<\/th><th>Entry processes<\/th><th>Physical memory<\/th><th>What that means for crawler load<\/th><\/tr><\/thead><tbody><tr><td>Bronze<\/td><td>20<\/td><td>1 GB<\/td><td>Fine for a cached content site; tight if search URLs are crawlable<\/td><\/tr><tr><td>Silver<\/td><td>30<\/td><td>2 GB<\/td><td>Comfortable headroom once articles are served from cache<\/td><\/tr><tr><td>WooStart<\/td><td>40<\/td><td>3 GB<\/td><td>Sized for cart and checkout paths, which are uncached by design<\/td><\/tr><tr><td>Gold<\/td><td>50<\/td><td>4 GB<\/td><td>Room for a large uncached surface alongside human peak<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">AHosting shared plan entry-process allocations, verified against the live server configuration on 31 August 2026. The ceiling is shared between crawler and human requests.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">In other words, bot traffic consumes entry processes only where your cache does not cover it, and that ceiling is what the plan tiers set. If your uncached surface is small, crawler load is not the reason to move tier. If it is large and genuinely necessary, the <a href=\"https:\/\/www.ahosting.net\/wordpress-hosting.html\">WordPress hosting tiers<\/a> are where that headroom comes from, and the estimator further down turns your own numbers into the comparison.<\/p>\n\n\n\n<h2 id=\"aioseo-why-robots-txt-does-not-fix-the-biggest-ai-crawler-number\" class=\"wp-block-heading\">Why robots.txt Does Not Fix the Biggest AI Crawler Number<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">That said, none of the above means robots.txt is useless. It means it is a policy instrument that is widely mistaken for a load control, and the difference shows up precisely where the volume is.<\/p>\n\n\n\n<h3 id=\"aioseo-what-perplexitys-own-documentation-says\" class=\"wp-block-heading\">What Perplexity&#8217;s Own Documentation Says<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Fortunately, this does not require inference, because the vendor states it plainly. Perplexity&#8217;s crawler page says that since a user requested the fetch, that fetcher generally ignores robots.txt rules. Perplexity-User is 38,743 of our 65,044 fetches.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Similarly, the pattern is not unique to one company. Live-browsing agents across the sector are framed as acting for a person rather than as autonomous crawlers, and the exclusion protocol has never claimed authority over a request a human initiated. <a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/google-common-crawlers\" target=\"_blank\" rel=\"noopener\">Google&#8217;s own crawler reference<\/a> makes the same three-way split between common crawlers, special-case crawlers and user-triggered fetchers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Consequently, a robots.txt file that disallows every AI agent you can name will still leave the majority of this traffic arriving. Believing otherwise is the specific failure mode this article exists to correct, and it is expensive only in the sense that it stops people looking at the thing that would actually help.<\/p>\n\n\n\n<h3 id=\"aioseo-what-robots-txt-still-does-control\" class=\"wp-block-heading\">What robots.txt Still Does Control<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">However, the 18,168 indexing fetches are a different matter, and there the file works exactly as documented. Blocking a training crawler stops that crawler. Allowing a search crawler is what keeps you eligible to be cited, and OpenAI notes that a site opted out of OAI-SearchBot will not appear in ChatGPT search answers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example, AHosting.net runs the split deliberately rather than by default. The live file allows GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Applebot-Extended, and disallows CCBot, Bytespider, TikTokBot, anthropic-ai, cohere-ai, meta-externalagent and Amazonbot. Agents that cite and link are welcome; agents that only harvest are not.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Additionally, the file carries a Content-Signal declaration reading search=yes, ai-train=no, ai-input=yes \u2014 a machine-readable statement that the site may be indexed for search and used to answer a live question, but not used as training data. Notably, that is a statement of terms rather than an enforcement mechanism, which is the right way to think about the whole file.<\/p>\n\n\n\n<h2 id=\"aioseo-ai-crawler-traffic-vs-a-bot-flood-which-problem-do-you-have\" class=\"wp-block-heading\">AI Crawler Traffic vs. a Bot Flood: Which Problem Do You Have?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In contrast to everything above, some sites really are being hurt by automated traffic \u2014 and when they are, an AI crawler is usually not what is doing it. The two present differently enough that a few minutes in the access log settles which one you have.<\/p>\n\n\n\n<figure class=\"wp-block-table ah-ladder\"><table><thead><tr><th>Signal<\/th><th>Legitimate AI crawler traffic<\/th><th>Abusive bot flood<\/th><\/tr><\/thead><tbody><tr><td>Target<\/td><td>Article and category pages, spread across the site<\/td><td>One endpoint hit repeatedly \u2014 login, XML-RPC, search<\/td><\/tr><tr><td>Rate<\/td><td>Steady; 65,044 fetches over 28 days averaged 1.6 a minute<\/td><td>Bursty; thousands of requests an hour against one URL<\/td><\/tr><tr><td>Identity<\/td><td>Declared agent with vendor-published IP ranges you can verify<\/td><td>Spoofed or absent agent, unverifiable addresses<\/td><\/tr><tr><td>Cache behavior<\/td><td>Mostly served from cache, so mostly free<\/td><td>Deliberately targets uncached and authenticated paths<\/td><\/tr><tr><td>Correct response<\/td><td>Audit the uncached surface; set policy in robots.txt<\/td><td>Rate-limit or block at the edge<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">How to tell an AI crawler apart from an abusive bot flood, from AHosting support cases and the access-log patterns behind them.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Therefore the diagnosis changes the action completely. An XML-RPC flood is handled the way <a href=\"https:\/\/www.ahosting.net\/blog\/stop-xml-rpc-bot-flood\/\">our guide to stopping one<\/a> describes, at the edge and quickly. AI crawler load is handled by looking at what you leave uncached, which is slower, duller and permanent.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Above all, verify before you act. Every major vendor publishes the IP ranges its agents use, so an agent string claiming to be a well-known crawler can be confirmed or rejected rather than assumed \u2014 and traffic impersonating a crawler is by definition the second column, not the first.<\/p>\n\n\n\n<h2 id=\"aioseo-what-ahosting-actually-ships-for-ai-crawler-load\" class=\"wp-block-heading\">What AHosting Actually Ships for AI Crawler Load<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Specifically, three things carry this on our stack, and none of them was built for AI. Server-level LiteSpeed caching removes the entry-process cost from every page it covers. The per-tier entry-process allocation sets the ceiling that the uncached remainder competes for. And the access logs are the only place any of this can be measured rather than guessed at.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Moreover, the estimator below is the arithmetic, not a product pitch. Put in your own daily fetch count, your honest cache-miss share and your tier, and it returns how much of that ceiling the crawler load is actually holding. Most sites that run it discover the answer is almost none, which is the useful finding.<\/p>\n\n\n\n<style>\n.ahacl-tool{background:#0f172a;border-radius:12px;padding:24px;margin:28px 0;color:#e2e8f0;font-size:16px}\n.ahacl-tool h3{color:#ffffff;margin:0 0 8px;font-size:21px;line-height:1.3}\n.ahacl-lede{color:#cbd5e1;margin:0 0 18px;font-size:14.5px;line-height:1.55}\n.ahacl-tool label{display:block;color:#eef3ff;font-size:14px;font-weight:600;margin:0 0 6px}\n.ahacl-tool input,.ahacl-tool select{width:100%;max-width:320px;padding:10px 12px;border-radius:8px;border:1px solid #475569;background:#1e293b;color:#ffffff;font-size:16px}\n.ahacl-row{margin:0 0 16px}\nbutton.ahacl-btn.wp-element-button,a.ahacl-cta.wp-element-button{display:inline-block;background:#2563eb !important;color:#ffffff !important;text-decoration:none !important;border:0;border-radius:8px;padding:11px 20px;font-size:15px;font-weight:600;cursor:pointer}\n.ahacl-out{margin:18px 0 0}\n.ahacl-hd{color:#eef3ff;font-size:13px;font-weight:700;text-transform:uppercase;letter-spacing:.04em;margin:0 0 10px}\n.ahacl-item{display:flex;justify-content:space-between;gap:12px;padding:9px 12px;border-radius:7px;background:#1e293b;margin:0 0 6px;font-size:15px}\n.ahacl-item span:first-child{color:#e2e8f0}\n.ahacl-item span:last-child{color:#93c5fd;font-weight:700;white-space:nowrap}\n.ahacl-verdict{background:#020617;border:1px solid #334155;border-radius:7px;padding:12px 14px;margin:10px 0 0;color:#86efac;font-size:14.5px;line-height:1.5}\n.ahacl-note{color:#cbd5e1;font-size:13.5px;line-height:1.55;margin:14px 0 0}\n<\/style>\n<div class=\"ahacl-tool\">\n  <h3>AI Crawler Load Estimator<\/h3>\n  <p class=\"ahacl-lede\">Give it your daily AI fetch count, the share of those that miss your cache, and your plan tier. It returns the entry processes that load actually occupies at peak, and whether the tier has room for it.<\/p>\n  <div class=\"ahacl-row\">\n    <label for=\"ahacl-fetches\">AI crawler fetches per day<\/label>\n    <input id=\"ahacl-fetches\" type=\"number\" min=\"0\" max=\"500000\" step=\"100\" value=\"2323\">\n  <\/div>\n  <div class=\"ahacl-row\">\n    <label for=\"ahacl-miss\">Share of those fetches that miss the cache<\/label>\n    <select id=\"ahacl-miss\">\n      <option value=\"2\">2% &#8211; articles cached, little else exposed<\/option>\n      <option value=\"10\" selected>10% &#8211; some feeds and search URLs reachable<\/option>\n      <option value=\"25\">25% &#8211; faceted or parameterized URLs crawlable<\/option>\n      <option value=\"60\">60% &#8211; little or no server-level caching<\/option>\n    <\/select>\n  <\/div>\n  <div class=\"ahacl-row\">\n    <label for=\"ahacl-tier\">Plan tier<\/label>\n    <select id=\"ahacl-tier\">\n      <option value=\"20\">Bronze &#8211; 20 entry processes<\/option>\n      <option value=\"30\" selected>Silver &#8211; 30 entry processes<\/option>\n      <option value=\"40\">WooStart &#8211; 40 entry processes<\/option>\n      <option value=\"50\">Gold &#8211; 50 entry processes<\/option>\n    <\/select>\n  <\/div>\n  <button type=\"button\" class=\"ahacl-btn wp-element-button\" data-ahacl=\"estimate\">Estimate the load<\/button>\n  <div id=\"ahacl-out\" class=\"ahacl-out\" role=\"status\" aria-live=\"polite\"><\/div>\n  <p class=\"ahacl-note\">Arithmetic over your own numbers, not a prediction. It assumes a 400 ms average dynamic response and a peak hour carrying four times the average rate, and it counts crawler load only &#8211; your human traffic occupies the same ceiling alongside it.<\/p>\n  <p class=\"ahacl-row\" style=\"margin:16px 0 0\"><a class=\"ahacl-cta wp-element-button\" href=\"https:\/\/www.ahosting.net\/web-hosting.html\">Compare the shared hosting tiers<\/a><\/p>\n<\/div>\n<script>\n(function(){\n  document.addEventListener('DOMContentLoaded', function(){\n    var root = document.querySelector('.ahacl-tool');\n    if (!root) { return; }\n    var out = document.getElementById('ahacl-out');\n    if (!out) { return; }\n    var fetchField = document.getElementById('ahacl-fetches');\n    if (!fetchField) { return; }\n    var missField = document.getElementById('ahacl-miss');\n    if (!missField) { return; }\n    var tierField = document.getElementById('ahacl-tier');\n    if (!tierField) { return; }\n    function row(labelText, valueText) {\n      return '<div class=\"ahacl-item\"><span>' + labelText + '<\/span><span>' + valueText + '<\/span><\/div>';\n    }\n    function estimate() {\n      var daily = parseInt(fetchField.value, 10);\n      if (isNaN(daily)) { daily = 0; }\n      if (daily < 0) { daily = 0; }\n      var missPct = parseInt(missField.value, 10);\n      var cap = parseInt(tierField.value, 10);\n      var uncached = Math.round(daily * missPct \/ 100);\n      var perHour = uncached \/ 24;\n      var peakHour = perHour * 4;\n      var concurrent = peakHour * 0.4 \/ 3600;\n      var shown = Math.round(concurrent * 100) \/ 100;\n      var headroom = Math.round((cap - concurrent) * 10) \/ 10;\n      var pct = Math.round(concurrent \/ cap * 1000) \/ 10;\n      var html = '<div class=\"ahacl-hd\">What that load occupies<\/div>';\n      html += row('Fetches reaching PHP each day', uncached.toLocaleString());\n      html += row('Peak-hour dynamic requests', Math.round(peakHour).toLocaleString());\n      html += row('Entry processes held at peak', shown.toFixed(2));\n      html += row('Share of the tier ceiling', pct.toFixed(1) + '%');\n      html += row('Entry processes left for visitors', headroom.toFixed(1));\n      var verdict = '';\n      if (pct < 5) {\n        verdict = 'Crawler load is not your constraint. At this cache-miss rate the fetches occupy a fraction of one entry process, so a larger tier would buy nothing against this traffic. Spend the effort on human peak instead.';\n      }\n      if (pct >= 5) {\n        if (pct < 25) {\n          verdict = 'Visible but not binding. Worth reducing the uncached surface before considering a tier change, because the same edit that removes those paths also removes the cost.';\n        }\n      }\n      if (pct >= 25) {\n        verdict = 'Crawler load is now competing with your visitors for the same ceiling. Audit what is reachable and uncached first; if the surface is genuinely necessary, the tier is the thing that has to move.';\n      }\n      html += '<div class=\"ahacl-verdict\">' + verdict + '<\/div>';\n      out.innerHTML = html;\n    }\n    root.addEventListener('click', function(e) {\n      var t = e.target;\n      if (!t) { return; }\n      var action = t.getAttribute('data-ahacl');\n      if (!action) { return; }\n      if (action === 'estimate') { estimate(); }\n    });\n    estimate();\n  });\n})();\n<\/script>\n\n\n\n<p class=\"wp-block-paragraph\">Ultimately, where the estimator says the load is binding, the constraint is real and the options are the ones it names: reduce what is reachable and uncached, or raise the ceiling. For a site whose uncached surface is genuinely necessary \u2014 a large store, a busy membership area, an application rather than a publication \u2014 that ceiling stops being shared at all on <a href=\"https:\/\/www.ahosting.net\/vps-hosting.html\">a VPS plan<\/a>. Where the pressure is coming from human concurrency instead, <a href=\"https:\/\/www.ahosting.net\/blog\/wordpress-php-workers-503-errors-2026\/\">the PHP worker calculation<\/a> sizes the same ceiling from the other direction.<\/p>\n\n\n\n<h2 id=\"aioseo-a-practical-checklist-for-ai-crawler-traffic\" class=\"wp-block-heading\">A Practical Checklist for AI Crawler Traffic<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Finally, work through this in order. Every item is answerable from your own logs and your own configuration, and the sequence matters \u2014 a policy decision made before the measurement is a preference rather than a diagnosis.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Access logs grouped by user agent over a fixed window, so the fetch count is measured rather than sensed.<\/li>\n\n\n\n<li>User-triggered agents separated from indexing agents before any total is quoted.<\/li>\n\n\n\n<li>Referred visits from assistants counted over the same window, so the return side is known too.<\/li>\n\n\n\n<li>Cache hit rate established for article pages, since that is what decides whether the load costs anything.<\/li>\n\n\n\n<li>Uncached surface listed explicitly: search URLs, faceted parameters, feeds, calendar archives, ajax endpoints.<\/li>\n\n\n\n<li>Anything on that list that does not need to be crawlable removed from the crawl surface.<\/li>\n\n\n\n<li>Entry-process ceiling for the current tier known, and the peak-hour crawler share estimated against it.<\/li>\n\n\n\n<li>robots.txt treated as a policy statement about indexing and training, never as a load control.<\/li>\n\n\n\n<li>Agent identity verified against published vendor IP ranges before anything is blocked.<\/li>\n\n\n\n<li>The decision revisited quarterly, because agent behavior and volumes are changing faster than the standards are.<\/li>\n<\/ul>\n\n\n\n<div class=\"ah-infographic\">\n<svg viewBox=\"0 0 820 400\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" role=\"img\" aria-label=\"Diagram of 65,044 AI crawler fetches recorded on AHosting over 28 days, split into 46,876 user-triggered fetches that robots.txt does not reliably stop and 18,168 indexing crawls that it does, then routed through the cache to show that only uncached requests consume an entry process.\" style=\"width:100%;height:auto;font-family:system-ui,-apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif\">\n  <title>Where 65,044 AI crawler fetches go, and which of them cost an entry process<\/title>\n  <desc>The 28-day AHosting total of 65,044 AI crawler fetches divides into 46,876 user-triggered fetches from Perplexity-User, ChatGPT-User and Claude-User, which vendors state may ignore robots.txt, and 18,168 indexing crawls which obey it. Both halves then meet the cache: a request served from the server-level cache starts no dynamic script and consumes no entry process, while a request to an uncached path consumes the same entry process a human visitor would.<\/desc>\n  <rect x=\"0\" y=\"0\" width=\"820\" height=\"400\" fill=\"#f8fafc\"\/>\n  <text x=\"24\" y=\"34\" font-size=\"19\" font-weight=\"700\" fill=\"#0f172a\">65,044 AI crawler fetches, 28 days, one hosting network<\/text>\n  <text x=\"24\" y=\"56\" font-size=\"13\" fill=\"#475569\">The split on the left decides what you can control. The gate on the right decides what it costs.<\/text>\n  <rect x=\"24\" y=\"86\" width=\"250\" height=\"86\" rx=\"8\" fill=\"#1e3a8a\"\/>\n  <text x=\"40\" y=\"118\" font-size=\"15\" font-weight=\"700\" fill=\"#ffffff\">46,876 user-triggered<\/text>\n  <text x=\"40\" y=\"140\" font-size=\"12\" fill=\"#bfdbfe\">Perplexity-User, ChatGPT-User,<\/text>\n  <text x=\"40\" y=\"157\" font-size=\"12\" fill=\"#bfdbfe\">Claude-User &#8211; may ignore robots.txt<\/text>\n  <rect x=\"24\" y=\"186\" width=\"250\" height=\"86\" rx=\"8\" fill=\"#334155\"\/>\n  <text x=\"40\" y=\"218\" font-size=\"15\" font-weight=\"700\" fill=\"#ffffff\">18,168 indexing<\/text>\n  <text x=\"40\" y=\"240\" font-size=\"12\" fill=\"#cbd5e1\">PerplexityBot, GPTBot, ClaudeBot,<\/text>\n  <text x=\"40\" y=\"257\" font-size=\"12\" fill=\"#cbd5e1\">Amazonbot &#8211; robots.txt applies<\/text>\n  <path d=\"M274 129 L354 175\" stroke=\"#94a3b8\" stroke-width=\"2\" fill=\"none\"\/>\n  <path d=\"M274 229 L354 185\" stroke=\"#94a3b8\" stroke-width=\"2\" fill=\"none\"\/>\n  <rect x=\"354\" y=\"152\" width=\"118\" height=\"56\" rx=\"8\" fill=\"#eff6ff\" stroke=\"#2563eb\" stroke-width=\"2\"\/>\n  <text x=\"372\" y=\"177\" font-size=\"14\" font-weight=\"700\" fill=\"#1e3a8a\">Cached?<\/text>\n  <text x=\"368\" y=\"196\" font-size=\"11.5\" fill=\"#1e40af\">server-level cache<\/text>\n  <path d=\"M472 168 L546 128\" stroke=\"#16a34a\" stroke-width=\"2\" fill=\"none\"\/>\n  <path d=\"M472 192 L546 232\" stroke=\"#b91c1c\" stroke-width=\"2\" fill=\"none\"\/>\n  <rect x=\"546\" y=\"98\" width=\"250\" height=\"72\" rx=\"8\" fill=\"#dcfce7\" stroke=\"#16a34a\"\/>\n  <text x=\"562\" y=\"126\" font-size=\"15\" font-weight=\"700\" fill=\"#14532d\">Yes &#8211; 0 entry processes<\/text>\n  <text x=\"562\" y=\"148\" font-size=\"12\" fill=\"#166534\">No dynamic script runs. The fetch costs<\/text>\n  <text x=\"562\" y=\"164\" font-size=\"12\" fill=\"#166534\">the plan nothing measurable.<\/text>\n  <rect x=\"546\" y=\"198\" width=\"250\" height=\"72\" rx=\"8\" fill=\"#fee2e2\" stroke=\"#b91c1c\"\/>\n  <text x=\"562\" y=\"226\" font-size=\"15\" font-weight=\"700\" fill=\"#7f1d1d\">No &#8211; 1 entry process<\/text>\n  <text x=\"562\" y=\"248\" font-size=\"12\" fill=\"#991b1b\">Search URLs, faceted parameters, admin-<\/text>\n  <text x=\"562\" y=\"264\" font-size=\"12\" fill=\"#991b1b\">ajax, feeds. Same slot a visitor uses.<\/text>\n  <rect x=\"24\" y=\"292\" width=\"772\" height=\"1\" fill=\"#e2e8f0\"\/>\n  <rect x=\"24\" y=\"306\" width=\"772\" height=\"54\" rx=\"8\" fill=\"#eff6ff\" stroke=\"#bfdbfe\"\/>\n  <text x=\"40\" y=\"330\" font-size=\"15\" fill=\"#1e3a8a\">The bill is not the fetch count. The bill is the uncached share of the fetch count.<\/text>\n  <text x=\"40\" y=\"350\" font-size=\"12\" fill=\"#1e40af\">Which is why the fix is a caching audit, not a robots.txt edit &#8211; and why the two are often confused.<\/text>\n  <text x=\"24\" y=\"384\" font-size=\"11.5\" fill=\"#64748b\">AHosting server access logs, 4 August to 31 August 2026. Referred visits back over the same window: 74 from ChatGPT, 5 from Gemini.<\/text>\n<\/svg>\n<\/div>\n\n\n\n<p class=\"wp-block-paragraph\">In fact, the discipline this asks for is modest. Count what arrives, split it by whether a person asked for it, find out how much of it reaches PHP, and only then decide anything. On the network measured here that sequence turned a number that looks like a crisis into a caching question with a short answer, and it will do the same for most sites that run it honestly.<\/p>\n\n\n\n<h2 id=\"faq-ai-crawler-traffic-hosting-cost\" class=\"wp-block-heading\">Frequently Asked Questions About AI Crawler Traffic<\/h2>\n\n\n\n<h3 id=\"faq-ai-crawler-cost-2026\" class=\"wp-block-heading\">Do AI crawler visits actually cost me anything on shared hosting in 2026?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Typically, only where the request reaches PHP. A page served from the server-level cache never starts a dynamic script, so it consumes no entry process and costs the plan nothing measurable. The cost appears on the paths caching does not cover, which is a much smaller and much more specific list than the raw fetch count suggests.<\/p>\n\n\n\n<h3 id=\"faq-ai-crawler-vs-search-crawler\" class=\"wp-block-heading\">AI crawler vs search engine crawler: what is the difference for my server?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">In practice, the load behaves the same and the control does not. A search crawler is building an index and comes back on its own schedule, so it obeys robots.txt and it can be slowed. A user-triggered AI crawler arrives because a person asked a question a moment ago, and several vendors state that this class of fetcher does not treat robots.txt as binding.<\/p>\n\n\n\n<h3 id=\"faq-block-ai-crawlers-robots\" class=\"wp-block-heading\">Can I block AI crawler traffic with robots.txt, and does it work?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Specifically, it works for the indexing half and not reliably for the other. Perplexity&#8217;s own documentation says its user-triggered fetcher generally ignores robots.txt rules because a person requested the page. Blocking is therefore a policy statement about training and indexing rather than a load control, and treating it as a load control is where most advice on this topic goes wrong.<\/p>\n\n\n\n<h3 id=\"faq-ahosting-ai-crawler-policy-2026\" class=\"wp-block-heading\">What is the AHosting robots.txt policy on AI crawler access in 2026?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Notably, AHosting allows the assistants that cite and link, and blocks the ones that only harvest. The live file explicitly allows GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Applebot-Extended, and disallows CCBot, Bytespider, TikTokBot, anthropic-ai, cohere-ai, meta-externalagent and Amazonbot. It also carries a Content-Signal line reading search=yes, ai-train=no, ai-input=yes.<\/p>\n\n\n\n<h3 id=\"faq-user-triggered-vs-indexing\" class=\"wp-block-heading\">User-triggered fetches vs indexing crawls: which one should I worry about?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Ultimately, the user-triggered half, because it is both larger and less controllable. Across 28 days on AHosting the three user-triggered agents accounted for 46,876 of 65,044 total fetches. That is the portion that arrives in response to a real question, cannot be scheduled away, and is the only portion with any chance of sending a reader back.<\/p>\n\n\n\n<h3 id=\"faq-entry-processes-bots\" class=\"wp-block-heading\">How many entry processes does an AI crawler fetch consume on a cached page?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Fortunately, none. CloudLinux defines an entry process as a concurrent connection to a dynamic script, and a page returned by the server-level cache never invokes one. Where the fetch lands on an uncached path, though, it consumes exactly the same entry process a human visitor would, and it competes for the same ceiling.<\/p>\n\n\n\n<h3 id=\"faq-ahosting-plan-tiers-2026\" class=\"wp-block-heading\">Which AHosting plan tier gives enough entry processes for heavy AI crawler traffic in 2026?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Accordingly, the answer depends on your uncached surface, not on the fetch count. The shared tiers allocate 20, 30, 40 and 50 entry processes for Bronze, Silver, WooStart and Gold respectively, verified against the live server configuration. A site whose cache covers its article pages rarely needs a larger tier for crawler load alone; a site with heavy search or faceted URLs often does.<\/p>\n\n\n\n<h3 id=\"faq-ai-crawler-referrals\" class=\"wp-block-heading\">How many visitors does AI crawler traffic actually send back to a hosting site?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Indeed, far fewer than the read volume implies. The same 28-day window that recorded 65,044 AI fetches on AHosting recorded 74 referred visits from ChatGPT and 5 from Gemini. That ratio is the honest starting point for any conversation about whether to allow, throttle or block, and it is a number most sites have never worked out.<\/p>\n\n\n\n<h3 id=\"faq-measure-ai-crawler-load\" class=\"wp-block-heading\">How do I measure AI crawler load in my own server access logs?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">For example, group raw access-log requests by user agent over a fixed window, then separate the user-triggered agents from the indexing ones before you total anything. The vendors publish the exact agent strings and their IP ranges, so verification is possible rather than guesswork. Measuring first is what stops a policy decision being made on an impression.<\/p>\n\n\n\n<h3 id=\"faq-ahosting-bot-flood-difference\" class=\"wp-block-heading\">How does AHosting tell an AI crawler apart from an ordinary bot flood?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">By contrast, a flood concentrates on one endpoint and a crawler spreads across content. Automated abuse hammers a login form, an XML-RPC endpoint or a search parameter thousands of times an hour from unverifiable addresses. Legitimate AI crawler traffic identifies itself, publishes its address ranges, and reads article pages, which is why the two need different responses.<\/p>\n\n\n\n<script>\n(function(){\n  document.addEventListener('DOMContentLoaded', function(){\n    var allH3s = document.querySelectorAll('h3.wp-block-heading');\n    var inFaq = false;\n    for (var i = 0; i < allH3s.length; i++) {\n      var h3 = allH3s[i];\n      var prev = h3.previousElementSibling;\n      if (prev) {\n        if (prev.tagName === 'H2') {\n          var prevId = prev.getAttribute('id');\n          if (prevId) {\n            if (prevId.indexOf('faq-') === 0) {\n              inFaq = true;\n            } else {\n              inFaq = false;\n            }\n          }\n        }\n      }\n      if (inFaq) {\n        initToggle(h3);\n      }\n    }\n    function initToggle(h3) {\n      var answer = h3.nextElementSibling;\n      if (!answer) { return; }\n      if (answer.tagName !== 'P') { return; }\n      var chev = document.createElement('span');\n      chev.className = 'ahfaq-chev ahfaq-chev-closed';\n      chev.setAttribute('aria-hidden', 'true');\n      h3.appendChild(chev);\n      h3.setAttribute('tabindex', '0');\n      h3.setAttribute('aria-expanded', 'false');\n      answer.classList.add('ahfaq-collapsed');\n      h3.addEventListener('click', function(){ doToggle(h3, answer, chev); });\n      h3.addEventListener('keydown', function(e){\n        if (e.key === 'Enter') { e.preventDefault(); doToggle(h3, answer, chev); }\n        if (e.key === ' ') { e.preventDefault(); doToggle(h3, answer, chev); }\n      });\n    }\n    function doToggle(h3, answer, chev) {\n      var isOpen = h3.getAttribute('aria-expanded') === 'true';\n      if (isOpen) {\n        answer.classList.remove('ahfaq-open');\n        answer.classList.add('ahfaq-collapsed');\n        h3.setAttribute('aria-expanded', 'false');\n        chev.classList.add('ahfaq-chev-closed');\n        chev.classList.remove('ahfaq-chev-open');\n      } else {\n        answer.classList.remove('ahfaq-collapsed');\n        answer.classList.add('ahfaq-open');\n        h3.setAttribute('aria-expanded', 'true');\n        chev.classList.remove('ahfaq-chev-closed');\n        chev.classList.add('ahfaq-chev-open');\n      }\n    }\n  });\n})();\n<\/script>\n","protected":false},"excerpt":{"rendered":"<p>TL;DR AHosting logged 65,044 AI crawler fetches in 28 days against 298 Google clicks. Most of that load costs nothing, because a cached page consumes no entry process \u2014 and most of it cannot be blocked with robots.txt either, because 72% of it is user-triggered. Every hosting blog now carries advice about AI crawler traffic, [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":1230,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"footnotes":""},"categories":[151],"tags":[161,163,84,104,165,267,46],"class_list":["post-1229","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hosting-guides","tag-aeo","tag-ai-crawlers","tag-entry-processes","tag-litespeed-cache","tag-robots-txt","tag-server-logs","tag-wordpress-hosting"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.1.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"We logged 65,044 AI crawler fetches in 28 days. Most cost nothing, and most cannot be blocked with robots.txt. Here is what the measurement actually shows.\" \/>\n\t<meta name=\"robots\" content=\"max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n\t<meta name=\"author\" content=\"Matt Chrust\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.1.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"Blog Home | AHosting Blog\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"AI Crawler Traffic: What 65,044 Fetches Cost Your Host\" \/>\n\t\t<meta property=\"og:description\" content=\"We logged 65,044 AI crawler fetches in 28 days. Most cost nothing, and most cannot be blocked with robots.txt. Here is what the measurement actually shows.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/www.ahosting.net\/blog\/wp-content\/uploads\/2026\/09\/ai-crawler-traffic-hosting-cost-ahosting.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/www.ahosting.net\/blog\/wp-content\/uploads\/2026\/09\/ai-crawler-traffic-hosting-cost-ahosting.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t\t<meta property=\"og:image:height\" content=\"675\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2026-09-04T17:27:07+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2026-09-04T19:50:11+00:00\" \/>\n\t\t<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n\t\t<meta name=\"twitter:site\" content=\"@ahostingdotnet\" \/>\n\t\t<meta name=\"twitter:title\" content=\"AI Crawler Traffic: What 65,044 Fetches Cost Your Host\" \/>\n\t\t<meta name=\"twitter:description\" content=\"We logged 65,044 AI crawler fetches in 28 days. Most cost nothing, and most cannot be blocked with robots.txt. Here is what the measurement actually shows.\" \/>\n\t\t<meta name=\"twitter:creator\" content=\"@ahostingdotnet\" \/>\n\t\t<meta name=\"twitter:image\" content=\"https:\/\/www.ahosting.net\/blog\/wp-content\/uploads\/2026\/09\/ai-crawler-traffic-hosting-cost-ahosting.jpg\" \/>\n\t\t<script type=\"application\/ld+json\" class=\"aioseo-schema\">\n\t\t\t{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"BlogPosting\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/#article\",\"name\":\"AI Crawler Traffic: What 65,044 Fetches Cost Your Host\",\"headline\":\"AI Crawler Traffic Is Now a Hosting Cost: 65,044 Fetches, Measured\",\"author\":{\"@type\":\"Person\",\"name\":\"Matt Chrust\",\"url\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/author\\\/matt-chrust\\\/\"},\"publisher\":{\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/#organization\"},\"image\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/ai-crawler-traffic-hosting-cost-ahosting.jpg\",\"width\":1200,\"height\":675,\"caption\":\"Assistants read the site 218 times for every visit Google search sent it. By Matt Chrust, Director of Business Development, AHosting.\"},\"datePublished\":\"2026-09-04T17:27:07+00:00\",\"dateModified\":\"2026-09-04T19:50:11+00:00\",\"inLanguage\":\"en-US\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/#webpage\"},\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/#webpage\"},\"articleSection\":\"Hosting Guides, AEO, AI crawlers, entry processes, litespeed cache, robots.txt, server logs, WordPress Hosting\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/#breadcrumblist\",\"itemListElement\":[{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog#listItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.ahosting.net\\\/blog\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/category\\\/hosting-guides\\\/#listItem\",\"name\":\"Hosting Guides\"}},{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/category\\\/hosting-guides\\\/#listItem\",\"position\":2,\"name\":\"Hosting Guides\",\"item\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/category\\\/hosting-guides\\\/\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/#listItem\",\"name\":\"AI Crawler Traffic Is Now a Hosting Cost: 65,044 Fetches, Measured\"},\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog#listItem\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/#listItem\",\"position\":3,\"name\":\"AI Crawler Traffic Is Now a Hosting Cost: 65,044 Fetches, Measured\",\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/category\\\/hosting-guides\\\/#listItem\",\"name\":\"Hosting Guides\"},\"item\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/\"}]},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/#organization\",\"name\":\"AHosting\",\"description\":\"AHosting Blog\",\"url\":\"https:\\\/\\\/www.ahosting.net\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/www.ahosting.net\\\/assets\\\/img\\\/ahosting-logo.svg\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/#organizationLogo\"},\"image\":{\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/#organizationLogo\"},\"sameAs\":[\"https:\\\/\\\/x.com\\\/ahostingdotnet\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/author\\\/matt-chrust\\\/#author\",\"url\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/author\\\/matt-chrust\\\/\",\"name\":\"Matt Chrust\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/#authorImage\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/795895edac1c44589f6c7f5e6bb79df405fbbaac15817bdd387ec57da61731ec?s=96&d=mm&r=g\",\"width\":96,\"height\":96,\"caption\":\"Matt Chrust\"}},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/#webpage\",\"url\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/\",\"name\":\"AI Crawler Traffic: What 65,044 Fetches Cost Your Host\",\"description\":\"We logged 65,044 AI crawler fetches in 28 days. Most cost nothing, and most cannot be blocked with robots.txt. Here is what the measurement actually shows.\",\"inLanguage\":\"en-US\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/#website\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/#breadcrumblist\"},\"author\":{\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/author\\\/matt-chrust\\\/#author\"},\"creator\":{\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/author\\\/matt-chrust\\\/#author\"},\"image\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/ai-crawler-traffic-hosting-cost-ahosting.jpg\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/#mainImage\",\"width\":1200,\"height\":675,\"caption\":\"Assistants read the site 218 times for every visit Google search sent it. By Matt Chrust, Director of Business Development, AHosting.\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/blog\\\/ai-crawler-traffic-hosting-cost\\\/#mainImage\"},\"datePublished\":\"2026-09-04T17:27:07+00:00\",\"dateModified\":\"2026-09-04T19:50:11+00:00\"},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/#website\",\"url\":\"https:\\\/\\\/www.ahosting.net\\\/\",\"name\":\"AHosting\",\"description\":\"AHosting Blog\",\"inLanguage\":\"en-US\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.ahosting.net\\\/#organization\"}}]}\n\t\t<\/script>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"AI Crawler Traffic: What 65,044 Fetches Cost Your Host","description":"We logged 65,044 AI crawler fetches in 28 days. Most cost nothing, and most cannot be blocked with robots.txt. Here is what the measurement actually shows.","canonical_url":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/","robots":"max-snippet:-1, max-image-preview:large, max-video-preview:-1","keywords":"","webmasterTools":{"miscellaneous":""},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"BlogPosting","@id":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/#article","name":"AI Crawler Traffic: What 65,044 Fetches Cost Your Host","headline":"AI Crawler Traffic Is Now a Hosting Cost: 65,044 Fetches, Measured","author":{"@type":"Person","name":"Matt Chrust","url":"https:\/\/www.ahosting.net\/blog\/author\/matt-chrust\/"},"publisher":{"@id":"https:\/\/www.ahosting.net\/#organization"},"image":{"@type":"ImageObject","url":"https:\/\/www.ahosting.net\/blog\/wp-content\/uploads\/2026\/09\/ai-crawler-traffic-hosting-cost-ahosting.jpg","width":1200,"height":675,"caption":"Assistants read the site 218 times for every visit Google search sent it. By Matt Chrust, Director of Business Development, AHosting."},"datePublished":"2026-09-04T17:27:07+00:00","dateModified":"2026-09-04T19:50:11+00:00","inLanguage":"en-US","mainEntityOfPage":{"@id":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/#webpage"},"isPartOf":{"@id":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/#webpage"},"articleSection":"Hosting Guides, AEO, AI crawlers, entry processes, litespeed cache, robots.txt, server logs, WordPress Hosting"},{"@type":"BreadcrumbList","@id":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/#breadcrumblist","itemListElement":[{"@type":"ListItem","@id":"https:\/\/www.ahosting.net\/blog#listItem","position":1,"name":"Home","item":"https:\/\/www.ahosting.net\/blog","nextItem":{"@type":"ListItem","@id":"https:\/\/www.ahosting.net\/blog\/category\/hosting-guides\/#listItem","name":"Hosting Guides"}},{"@type":"ListItem","@id":"https:\/\/www.ahosting.net\/blog\/category\/hosting-guides\/#listItem","position":2,"name":"Hosting Guides","item":"https:\/\/www.ahosting.net\/blog\/category\/hosting-guides\/","nextItem":{"@type":"ListItem","@id":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/#listItem","name":"AI Crawler Traffic Is Now a Hosting Cost: 65,044 Fetches, Measured"},"previousItem":{"@type":"ListItem","@id":"https:\/\/www.ahosting.net\/blog#listItem","name":"Home"}},{"@type":"ListItem","@id":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/#listItem","position":3,"name":"AI Crawler Traffic Is Now a Hosting Cost: 65,044 Fetches, Measured","previousItem":{"@type":"ListItem","@id":"https:\/\/www.ahosting.net\/blog\/category\/hosting-guides\/#listItem","name":"Hosting Guides"},"item":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/"}]},{"@type":"Organization","@id":"https:\/\/www.ahosting.net\/#organization","name":"AHosting","description":"AHosting Blog","url":"https:\/\/www.ahosting.net\/","logo":{"@type":"ImageObject","url":"https:\/\/www.ahosting.net\/assets\/img\/ahosting-logo.svg","@id":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/#organizationLogo"},"image":{"@id":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/#organizationLogo"},"sameAs":["https:\/\/x.com\/ahostingdotnet"]},{"@type":"Person","@id":"https:\/\/www.ahosting.net\/blog\/author\/matt-chrust\/#author","url":"https:\/\/www.ahosting.net\/blog\/author\/matt-chrust\/","name":"Matt Chrust","image":{"@type":"ImageObject","@id":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/#authorImage","url":"https:\/\/secure.gravatar.com\/avatar\/795895edac1c44589f6c7f5e6bb79df405fbbaac15817bdd387ec57da61731ec?s=96&d=mm&r=g","width":96,"height":96,"caption":"Matt Chrust"}},{"@type":"WebPage","@id":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/#webpage","url":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/","name":"AI Crawler Traffic: What 65,044 Fetches Cost Your Host","description":"We logged 65,044 AI crawler fetches in 28 days. Most cost nothing, and most cannot be blocked with robots.txt. Here is what the measurement actually shows.","inLanguage":"en-US","isPartOf":{"@id":"https:\/\/www.ahosting.net\/blog\/#website"},"breadcrumb":{"@id":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/#breadcrumblist"},"author":{"@id":"https:\/\/www.ahosting.net\/blog\/author\/matt-chrust\/#author"},"creator":{"@id":"https:\/\/www.ahosting.net\/blog\/author\/matt-chrust\/#author"},"image":{"@type":"ImageObject","url":"https:\/\/www.ahosting.net\/blog\/wp-content\/uploads\/2026\/09\/ai-crawler-traffic-hosting-cost-ahosting.jpg","@id":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/#mainImage","width":1200,"height":675,"caption":"Assistants read the site 218 times for every visit Google search sent it. By Matt Chrust, Director of Business Development, AHosting."},"primaryImageOfPage":{"@id":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/#mainImage"},"datePublished":"2026-09-04T17:27:07+00:00","dateModified":"2026-09-04T19:50:11+00:00"},{"@type":"WebSite","@id":"https:\/\/www.ahosting.net\/#website","url":"https:\/\/www.ahosting.net\/","name":"AHosting","description":"AHosting Blog","inLanguage":"en-US","publisher":{"@id":"https:\/\/www.ahosting.net\/#organization"}}]},"og:locale":"en_US","og:site_name":"Blog Home | AHosting Blog","og:type":"article","og:title":"AI Crawler Traffic: What 65,044 Fetches Cost Your Host","og:description":"We logged 65,044 AI crawler fetches in 28 days. Most cost nothing, and most cannot be blocked with robots.txt. Here is what the measurement actually shows.","og:url":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/","og:image":"https:\/\/www.ahosting.net\/blog\/wp-content\/uploads\/2026\/09\/ai-crawler-traffic-hosting-cost-ahosting.jpg","og:image:secure_url":"https:\/\/www.ahosting.net\/blog\/wp-content\/uploads\/2026\/09\/ai-crawler-traffic-hosting-cost-ahosting.jpg","og:image:width":1200,"og:image:height":675,"article:published_time":"2026-09-04T17:27:07+00:00","article:modified_time":"2026-09-04T19:50:11+00:00","twitter:card":"summary_large_image","twitter:site":"@ahostingdotnet","twitter:title":"AI Crawler Traffic: What 65,044 Fetches Cost Your Host","twitter:description":"We logged 65,044 AI crawler fetches in 28 days. Most cost nothing, and most cannot be blocked with robots.txt. Here is what the measurement actually shows.","twitter:creator":"@ahostingdotnet","twitter:image":"https:\/\/www.ahosting.net\/blog\/wp-content\/uploads\/2026\/09\/ai-crawler-traffic-hosting-cost-ahosting.jpg"},"aioseo_meta_data":{"post_id":"1229","title":"AI Crawler Traffic: What 65,044 Fetches Cost Your Host","description":"We logged 65,044 AI crawler fetches in 28 days. Most cost nothing, and most cannot be blocked with robots.txt. Here is what the measurement actually shows.","keywords":null,"keyphrases":{"focus":{"keyphrase":"AI crawler","score":0,"analysis":[]},"additional":[]},"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":"","og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"Article","isEnabled":true},"graphs":[]},"schema_type":"default","schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":"-1","robots_max_videopreview":"-1","robots_max_imagepreview":"large","priority":null,"frequency":"default","local_seo":null,"breadcrumb_settings":null,"limit_modified_date":false,"ai":{"faqs":[],"keyPoints":[],"schemas":[],"titles":[],"descriptions":[],"socialPosts":{"email":{"subject":"","preview":"","content":""},"linkedin":[],"twitter":[],"facebook":[],"instagram":[]}},"created":"2026-09-04 17:24:03","updated":"2026-09-04 19:52:24","seo_analyzer_scan_date":null,"focus_keyword":"AI crawler","additional_keywords":null,"truseo_locale":null},"aioseo_breadcrumb":"<div class=\"aioseo-breadcrumbs\"><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/www.ahosting.net\/blog\" title=\"Home\">Home<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/www.ahosting.net\/blog\/category\/hosting-guides\/\" title=\"Hosting Guides\">Hosting Guides<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\tAI Crawler Traffic Is Now a Hosting Cost: 65,044 Fetches, Measured\n\t\t<\/span><\/div>","aioseo_breadcrumb_json":[{"label":"Home","link":"https:\/\/www.ahosting.net\/blog"},{"label":"Hosting Guides","link":"https:\/\/www.ahosting.net\/blog\/category\/hosting-guides\/"},{"label":"AI Crawler Traffic Is Now a Hosting Cost: 65,044 Fetches, Measured","link":"https:\/\/www.ahosting.net\/blog\/ai-crawler-traffic-hosting-cost\/"}],"_links":{"self":[{"href":"https:\/\/www.ahosting.net\/blog\/wp-json\/wp\/v2\/posts\/1229","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ahosting.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ahosting.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ahosting.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ahosting.net\/blog\/wp-json\/wp\/v2\/comments?post=1229"}],"version-history":[{"count":3,"href":"https:\/\/www.ahosting.net\/blog\/wp-json\/wp\/v2\/posts\/1229\/revisions"}],"predecessor-version":[{"id":1234,"href":"https:\/\/www.ahosting.net\/blog\/wp-json\/wp\/v2\/posts\/1229\/revisions\/1234"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.ahosting.net\/blog\/wp-json\/wp\/v2\/media\/1230"}],"wp:attachment":[{"href":"https:\/\/www.ahosting.net\/blog\/wp-json\/wp\/v2\/media?parent=1229"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ahosting.net\/blog\/wp-json\/wp\/v2\/categories?post=1229"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ahosting.net\/blog\/wp-json\/wp\/v2\/tags?post=1229"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}