Short answer: if you want to show up in AI answers, allow the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot) and the user bots (ChatGPT-User, Claude-User). The training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) are a separate decision about whether your pages can be used to train models, and blocking them does not remove you from search. Below is what each name means, taken from each company's own documentation on 15 October 2026.
Why the names matter
When someone asks an AI assistant about your business, a few different programmes may touch your site, and they do different jobs. One builds a search index. One collects pages to train a model. One opens a single page because a person asked it to. Treating them as "the AI bot" is how people end up blocking the one that would have sent them customers.
The practical rule we use: separate "can it find me" from "can it learn from me". The first is the one that decides whether you can be mentioned. The second is a policy choice, and it is yours to make.
The crawlers, by company
| Company | Name in your logs | What it does | Follows robots.txt? |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Finds and links sites in ChatGPT search | Yes. OpenAI says changes take about 24 hours to take effect |
| OpenAI | GPTBot | Collects content that may be used to train models | Yes |
| OpenAI | ChatGPT-User | Opens a page when a person asks ChatGPT to | OpenAI says robots.txt rules may not apply, because a person initiated the visit |
| Anthropic | Claude-SearchBot | Improves the quality of Claude's search results | Yes |
| Anthropic | ClaudeBot | Collects content that may contribute to training | Yes, and Crawl-delay is supported |
| Anthropic | Claude-User | Fetches a page when a person asks Claude to | Yes |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity search. Perplexity says it is not used to train models | Yes |
| Perplexity | Perplexity-User | Visits a page when a person asks a question | Perplexity says it generally ignores robots.txt, because a person asked |
Googlebot | Google Search, which also feeds Google's AI search features | Yes | |
Google-Extended | A robots.txt token, not a separate bot. It controls use of your content for Gemini training and grounding | Yes. Google says it does not affect inclusion in Google Search | |
| Apple | Applebot | Crawls for Apple's search features such as Siri and Spotlight | Yes. With no Applebot rules it follows what you set for Googlebot |
| Apple | Applebot-Extended | A switch only. It does not crawl. Disallowing it opts your pages out of Apple's model training | Yes |
Which ones to allow
If you want customers to find you through AI answers: allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Applebot. Also leave ChatGPT-User, Claude-User and Perplexity-User alone. Those are the visits that happen the moment a person is asking about you.
If you do not want your pages used for training: disallow GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. This is where businesses differ, and neither answer is wrong. A shop that wants every possible mention might allow all of them. A publisher whose articles are the product may not.
Two caveats that people often miss. First, blocking training does not undo what was collected earlier. Second, a robots.txt rule is a request that the well-behaved bots follow. It is not a lock. If you need something kept out, put it behind a login.
Copy-paste robots.txt
Everything in, training included:
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/Found in AI answers, but not used for training:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/The search bots are not named in the second block, so they fall under the final group and are allowed. A bot follows the most specific group that names it, which is why the training bots can be shut out while everything else stays open.
Our own robots.txt names every crawler in the table above with Allow: /, and disallows /api/, /admin/ and one query string pattern. You can read it at devmarde.com/robots.txt.
What we saw when we tested our own site
On 14 October 2026 at 21:19 UTC we requested our home page with curl, once for each name in the table (minus the two robots.txt tokens, which do not make requests) plus a normal browser string. Every one returned 200 and the same 372,444 byte page, in between 0.9 and 2.9 seconds.
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
-A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" \
https://yourdomain.com/What this does not show. It sends a bot's name from our computer. It does not send the bot's network address, and a firewall that checks both could treat the real bot differently from our test. Each company publishes address lists so you can check: OpenAI, Anthropic and Perplexity. If your test fails, or you want to see the pattern for a 403, start with our guide to Cloudflare blocking AI bots.
You can run the same kind of test without a terminal on our free Crawler Tester.
Checking that it works
- Open your robots.txt in a browser and look for a group that names the crawler you care about. A bot with no named group follows
User-agent: *. - Run the curl line above for the search bots. You want
200and a full-size page. A403or429means something in front of your site is refusing, whatever robots.txt says. - Look in your server logs for the bot names over the next few weeks. A search bot that never shows up has not found you yet or is being blocked upstream.
- Run the free Crawl Audit check. It looks at your own domain for the same signals and more.
Access is step one of AI search optimization. If you would rather have someone check and fix it, that is what our AI visibility service does.
Questions people ask
What is ClaudeBot and should I block it?
ClaudeBot is Anthropic's crawler for content that may contribute to training its models. It follows robots.txt. Blocking it does not remove you from Claude's search, which uses Claude-SearchBot, or stop Claude-User opening a page when someone asks Claude about you. Whether to block it is a policy choice about training.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot builds the index behind ChatGPT search results. OpenAI treats them as independent settings, so you can disallow one and allow the other.
Is ChatGPT-User the same as GPTBot?
No. ChatGPT-User opens a page when a person asks ChatGPT to, for example to read a link they pasted. OpenAI says robots.txt rules may not apply to it, because a person started the visit.
Does blocking Google-Extended remove me from Google Search?
No. Google says it does not affect a site's inclusion in Google Search. It controls whether your content can be used for Gemini training and grounding.
Does PerplexityBot train models?
Perplexity says PerplexityBot is used to surface and link websites in search results and not to train generative models. Perplexity also says Perplexity-User generally ignores robots.txt, because a person asked for the page.
How long does a robots.txt change take to work?
OpenAI says about 24 hours for OAI-SearchBot. The other companies do not give a figure in the pages we read, so allow a few days and watch your logs.
Sources
- OpenAI, Overview of OpenAI crawlers, read 15 October 2026
- Anthropic, Does Anthropic crawl data from the web, read 15 October 2026
- Perplexity, Perplexity crawlers, read 15 October 2026
- Google Search Central, Google's common crawlers, read 15 October 2026
- Apple, About Applebot, read 15 October 2026
- Our own test: curl requests to devmarde.com, 14 October 2026, 21:19 UTC
Related
View all →Can ChatGPT read my website? Check it in 60 seconds
Four quick tests show whether ChatGPT, Claude and Perplexity can fetch and read your site: robots.txt, a crawler fetch, the raw HTML and your server logs. Free, no sign-up.
Cloudflare blocking ChatGPT? Fix the 403 challenge page
robots.txt can say yes while your firewall says no. How Cloudflare's AI bot controls, bot-fight modes and WAF rules block ChatGPT, Claude and Perplexity search bots by accident, how to spot it, and how to fix it.
AI search optimisation for UK small businesses: where to start
A five-day plan for a UK small business to become readable to AI answers: let the search bots in, show your company details the way GOV.UK expects, mark up your business, and build proof on profiles you do not own.
Find out whether AI answers can read your site.
Crawl Audit scores your own domain against the signals in this guide. The free check needs no sign-up; the full report is $7.89, one time.