Which AI crawlers can read your site: the crawlers that matter, and what blocking each one costs
In short
Most robots.txt files that block AI crawlers were written to keep content out of model training. Almost none of them were meant to remove the site from ChatGPT search results, and yet that is what many of them do. This page sorts the AI crawlers that decide commercial visibility by what each one is actually for, states what blocking it costs, and shows why the crawler that fails is usually stopped by the CDN rather than by robots.txt.
Simon WilhelmCo-founder
Simon is co-founder of scaile and works with enterprise teams on visibility in AI search, from the first audit to defensible Share of Voice numbers.
Key takeaways
Three vendors now run the same three-way split — a training crawler, a search crawler, and a user-initiated fetcher. OpenAI runs GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic runs ClaudeBot, Claude-SearchBot and Claude-User; Amazon runs Amazonbot, Amzn-SearchBot and Amzn-User. Perplexity, Apple and Google each do something different.
Blocking GPTBot keeps you out of training data. Blocking OAI-SearchBot means, in OpenAI's own words, your site "will not be shown in ChatGPT search answers, though can still appear as navigational links". These are entirely different decisions written on adjacent lines of the same file.
The user-initiated fetchers mostly do not obey robots.txt, and their vendors say so plainly. Anthropic is the exception: it states that all three of its bots honour robots.txt, Claude-User included.
Google-Extended is a control token with no crawler behind it. It governs Gemini training and grounding, and Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal".
robots.txt is only the first place to look. The CDN or WAF in front of the origin is the failure no robots.txt checker can see, and Perplexity's own documentation includes a section on unblocking its bots in Cloudflare and AWS WAF — a section a vendor only writes because customers keep hitting it.
Once access is confirmed, the remaining work is content. scaile writes the answers your buyers search for, checks every claim, and puts each one in front of your team for approval, within 14 days of onboarding.
There are roughly a dozen AI crawlers worth knowing about, and they split into three kinds: bots that collect training data, bots that build a search index the assistant queries at answer time, and bots that fetch one page because a user just asked something. Blocking a training crawler is a defensible editorial decision with no effect on today’s visibility. Blocking a search or fetch crawler removes you from answers your buyers are reading this afternoon, and that is almost never what the person editing robots.txt intended.
The more common failure is quieter still: robots.txt allows the bot, and the CDN blocks it anyway.
Getting access right is a day of work and a precondition, not the goal. Once the crawlers are through, whether you actually get cited depends on whether a complete, sourced answer to each of your buying questions exists on your site at all. That is the half most companies never start, and it is the half scaile runs as one managed pipeline: a named AI Search Strategist, research from your own knowledge base, every fact checked, your team approving before anything publishes, live in 14 days.
Which crawlers are actually asking for your pages?
Every row below comes from the operator’s own published documentation, except where the table says no such documentation exists. Where the documentation is ambiguous or contested, the table says so rather than guessing.
| Crawler | Run by | Training or live answers | Obeys robots.txt | What blocking it costs |
|---|---|---|---|---|
GPTBot | OpenAI | Training | Yes | Nothing today. Your content is excluded from future foundation-model training. |
OAI-SearchBot | OpenAI | Live answers | Yes | Not shown in ChatGPT search answers; can still appear as a navigational link. |
ChatGPT-User | OpenAI | Live, user-triggered | “robots.txt rules may not apply” | Little via robots.txt. Not used to decide Search appearance. |
ClaudeBot | Anthropic | Training | Yes | Your future material is excluded from Claude’s training sets. |
Claude-SearchBot | Anthropic | Live answers | Yes | Your content is not indexed for Claude search, reducing visibility in answers. |
Claude-User | Anthropic | Live, user-triggered | Yes (Anthropic’s stated policy) | Claude cannot retrieve your page when a user asks about you. |
PerplexityBot | Perplexity | Live answers, explicitly not training | Yes | You are not surfaced or linked in Perplexity results. |
Perplexity-User | Perplexity | Live, user-triggered | “generally ignores robots.txt” — and disputed | Little via robots.txt. See the dispute below. |
Google-Extended | Training and Gemini grounding | Control token only, no separate crawler | Gemini Apps cannot ground answers in your content. No effect on Google Search. | |
CCBot | Common Crawl | Training corpus | Yes | You leave the open crawl archive that many model builders draw on. |
Bytespider | ByteDance | Training | Disputed, no comparable public documentation | Little in Western markets: Cloudflare measured its crawl share falling from 22.8 to 2.9 percent between May 2024 and May 2025. |
Amazonbot | Amazon | Product improvement and AI training | Yes | Excluded from Amazon product improvement and from any Amazon model training the crawl may feed. |
Applebot-Extended | Apple | Training only | Control token only, no separate crawler | Apple foundation models are not trained on you. Spotlight, Siri and Safari search are unaffected. |
meta-externalagent | Meta | Training and indexing | Yes | Excluded from Meta’s foundation-model training. |
Two of those rows deserve their own sentence. Applebot-Extended is not a crawler; Apple’s documentation is explicit that even if you disallow it, “your content will remain discoverable through Spotlight, Siri, and Safari”. And Meta runs a separate Meta-WebIndexer for Meta AI search quality, which is the one that matters if you want to be cited there rather than merely trained on.
The table is a shortlist, not a registry. The same operators run further tokens that rarely change a decision on their own: Applebot itself, which does the crawling the Applebot-Extended token gates; Amzn-SearchBot and Amzn-User; meta-externalfetcher; and OAI-AdsBot. Each is documented in the sources at the foot of this page.
What is the difference between a training crawler and a citation crawler?
A training crawler takes a copy of your page into a dataset. Whatever it collects influences a model generation that ships months later, and nothing you publish today reaches an answer through that path any faster.
A citation crawler feeds a live index. When somebody asks ChatGPT a question that triggers a search, the system retrieves passages from that index and links to the source. That path is measured in days, not model generations, and it is where commercial visibility actually lives.
OpenAI states the split as clearly as anyone: “a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training”. That sentence is the whole decision. Refusing to be training material while remaining citable is a coherent position that many publishers hold deliberately. Blocking both because a template said User-agent: * is not a position, it is an accident.
The third kind, the user-initiated fetcher, is the one people misunderstand most. OpenAI writes that because ChatGPT-User actions “are initiated by a user, robots.txt rules may not apply”, and Perplexity says of Perplexity-User that “since a user requested the fetch, this fetcher generally ignores robots.txt rules”. Meta says the same of meta-externalfetcher, and Amazon of Amzn-User. If your goal was to stop assistants reading your pages at all, robots.txt was never the instrument. Anthropic is the outlier here and worth noting: it states that all of its bots, including Claude-User, respect robots.txt directives.
Is Perplexity-User really ignoring robots.txt?
Here the honest answer is that reporting is contested and you should not accept either side uncritically.
Perplexity’s documentation says what it says: Perplexity-User generally ignores robots.txt because a user requested the fetch. In August 2025, Cloudflare published a much harder claim — that Perplexity used undeclared user agents impersonating Chrome and rotated IP addresses across ASNs to reach content on brand-new test domains whose robots.txt disallowed everything. Cloudflare removed Perplexity from its Verified Bots programme. Perplexity rejected the finding publicly, called the report a publicity stunt, argued that Cloudflare had conflated user-driven requests with automated crawling, and denied the stealth crawler was theirs at all, attributing the traffic Cloudflare observed to BrowserBase, a third-party automated-browser service.
We have not reproduced either experiment, so we do not assert a verdict. What is safe to act on: robots.txt is a request, not a control. If you genuinely need a page not to be fetched, that requires authentication or a server-side rule, and it always did.
Why does robots.txt say yes while the crawler still gets a 403?
This is the finding that comes up most often in real audits, and no robots.txt checker will show it to you. The file permits the bot. The CDN, WAF or bot-management rule in front of the origin does not, because these crawlers arrive without cookies, without JavaScript execution and from data-centre address space, which is exactly the signature a generic bot rule is written to stop.
You do not have to take that on faith. Perplexity’s own documentation carries step-by-step instructions for allowing its bots in Cloudflare WAF and AWS WAF, combining user-agent matching with its published IP ranges. A vendor writes that section because customers keep hitting the problem, not because someone imagined it might happen.
Anthropic adds a related warning worth heeding in the other direction: blocking its bots by IP address “may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file”. A bot that cannot fetch robots.txt cannot obey it.
How do you verify access properly?
Three checks, in order, and only the first is the one most tools perform.
Read robots.txt as the bot reads it. Group order and wildcard handling matter, and a Disallow: / under User-agent: * applies to every crawler you did not name explicitly. The free Sitemap Visualiser reads your robots.txt and every sitemap behind it and draws which sections are reachable, which is faster than reasoning about precedence by hand.
Check your server logs for real hits. Configuration tells you what should happen; logs tell you what did. Filter on the tokens in the table above and look at the status codes, not just the presence of the request. A run of 403s from OAI-SearchBot is the whole diagnosis. If a bot has not appeared at all in ninety days, that is a finding too.
Verify the requests are genuine. User-agent strings are trivially forged, so confirm before you allowlist. OpenAI, Anthropic, Perplexity, Apple and Google all publish IP ranges as JSON; Amazon publishes its ranges as a page rather than a feed. Google and Apple additionally support reverse DNS. Matching user agent plus verified IP is the combination Perplexity itself recommends for WAF rules.
For the crawlers you want in, keep the robots.txt entries minimal and explicit:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Whether you also allow GPTBot, ClaudeBot, CCBot and Google-Extended is a separate, editorial question about training data. Answer it separately.
Does llms.txt change any of this?
No, and the data on that is now unambiguous. Ahrefs analysed 137,210 domains in 2026 and found that 97 percent of published llms.txt files received zero requests. Of the small remainder, no single AI bot outranked SEO audit tools, which were the largest single source of requests at 21.7 percent; every AI bot combined came to 19.5 percent, and the retrieval bots that actually feed answers to 1.1 percent. No major vendor documents reading the file — not OpenAI, not Anthropic, not Perplexity, and Google has said outright that Search does not use it.
Publishing one costs nothing and proves nothing. The file that actually governs crawler access is robots.txt, and the thing that actually governs whether you get cited is whether a complete, sourced answer to the question exists on a page a crawler can reach.
What do you do once access is clean?
Crawler access is a floor, not a strategy. It takes an afternoon, it is worth the afternoon, and then it is done. Cloudflare’s 2025 network data found that among top-10,000 domains with a robots.txt file, only about 14 percent had a directive naming AI bots at all. That does not mean the rest are open — a blanket Disallow: / and a WAF rule are both invisible to that count — but it does mean deliberate AI blocking is rarer than the discourse suggests.
What follows is harder and slower: the questions your buyers ask before they decide, each answered completely, with sources, on your own domain. That is the work scaile does. The engine finds the questions at the commercial end of your funnel, researches from your knowledge base and current sources, checks every claim, and puts each article in front of your team for approval. Published pieces are then measured and improved rather than left to age.
Start by seeing what the crawlers can currently reach with the Sitemap Visualiser, then check whether you are being named at all with the AI Visibility Check. If the answer is no and access is clean, the cause is one of the other three reasons ChatGPT does not mention a company.
FAQ
Is GPTBot blocked on my site?
Check robots.txt for a User-agent: GPTBot group and for a User-agent: * group with Disallow: /, which applies to GPTBot unless a more specific group overrides it. Then check server logs for actual GPTBot requests and their status codes, because a CDN can return 403 to a bot robots.txt permits.
Should I block GPTBot?
That depends on whether you object to your content being used for training. Blocking it has no effect on whether ChatGPT can cite you today — that is OAI-SearchBot’s job, and it is controlled separately. Many publishers deliberately block GPTBot and allow OAI-SearchBot.
Does blocking Google-Extended hurt my Google rankings?
No. Google states that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search”. It does stop Gemini Apps grounding answers in your content, which is a real cost, just not a Search one.
Can I stop AI assistants reading my pages entirely?
Not with robots.txt. The user-initiated fetchers — ChatGPT-User, Perplexity-User, meta-externalfetcher, Amzn-User — state in their own documentation that robots.txt may not apply to them. Anything stronger requires authentication or server-side blocking.
Why does a crawler checker say I am fine when I am still not cited?
Because those tools read robots.txt and nothing else. They cannot see a WAF rule, a rate limit, a geo-block, or a 403 in your access log, and they cannot tell you whether an answer to the question even exists on your site.
Does llms.txt help AI crawlers find my content?
No. In a study of 137,210 domains, 97 percent of llms.txt files were never requested, and no major AI vendor documents reading the file. A correct robots.txt and a complete sitemap do the job it claims to do.
Our crawler access is clean and we are still not cited. Who does the rest?
The rest is content, and it is a production problem rather than a technical one: every buying question needs a complete, sourced, current answer on your own site, and those answers have to keep being written. scaile does that as managed infrastructure, with a named AI Search Strategist, research from your knowledge base, fact-checking on every claim and your team approving before publication, live in 14 days. Building Radar doubled its qualified inbound leads in 90 days on it.
Sources
- OpenAI: Overview of OpenAI Crawlers. GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot: purpose, user-agent strings, robots.txt behaviour, and the statement that search opt-outs remove a site from ChatGPT search answers.
- Anthropic: Does Anthropic crawl data from the web?, updated April 2026. ClaudeBot, Claude-User and Claude-SearchBot, the statement that all three honour robots.txt, and the warning about IP-level blocking.
- Perplexity: Perplexity Crawlers. PerplexityBot and Perplexity-User, the “generally ignores robots.txt” statement, and the Cloudflare and AWS WAF allowlisting instructions.
- Cloudflare: Perplexity is using stealth, undeclared crawlers, August 2025. The contested claim about undeclared user agents and rotated IP ranges; Perplexity publicly rejected it.
- Google Search Central: Google common crawlers. Google-Extended as a control token without its own user agent, covering Gemini training and grounding, with no effect on Search.
- Google Search Central: AI features and your website. Indexing and snippet eligibility as the requirement for AI Overviews and AI Mode, and that no special files are needed.
- Common Crawl: CCBot. Operator, user-agent token and robots.txt blocking instructions.
- Amazon: Amazonbot. Amazonbot, Amzn-SearchBot and Amzn-User, and the statement that Amzn-User may not follow all robots.txt directives.
- Apple: About Applebot. Applebot-Extended as a training opt-out that leaves Spotlight, Siri and Safari discovery intact.
- Meta: Web crawlers. meta-externalagent, Meta-WebIndexer and meta-externalfetcher, including that the fetcher may bypass robots.txt.
- Cloudflare: From Googlebot to GPTBot, who’s crawling your site in 2025, July 2025. Crawl-share by bot, including Bytespider’s fall from 22.8 to 2.9 percent between May 2024 and May 2025, and that only about 14 percent of top-10,000 domains with a robots.txt file carry a directive naming AI bots.
- Ahrefs: llms.txt study, 137,210 domains, June 2026. 97 percent of llms.txt files received zero requests, and the category breakdown of the requests the remaining 3 percent did receive.
