Short answer: on 13 November 2026 we checked 20 sites that publish advice on AI search optimization. None of the 17 whose robots.txt we could read blocks the main AI search crawlers. One blocks Google's AI-training opt-out token, which is not a search bot. So "everyone is blocking ChatGPT" is not what we found. What we found instead: 3 sites would not answer a plain request at all, and 1 answered a normal browser but refused two OpenAI crawler names.
The numbers
| Check | Result (20 sites) |
|---|---|
| robots.txt readable | 17 of 20 |
| Name at least one AI crawler in robots.txt | 7 of 17 |
| Block GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot or PerplexityBot | 0 of 17 |
| Block Google-Extended (the AI training opt-out) | 1 of 17 |
| Serve a real llms.txt file | 12 of 20 |
| Home page answers 200 to a plain browser request | 17 of 20 |
| Home page answers differently to the OAI-SearchBot and GPTBot names than to a browser | 1 of 20 |
What stands out
- The robots.txt myth does not hold here. Not one of the 17 readable files disallows the search and answer crawlers. If your site is invisible to ChatGPT, the cause is probably somewhere other than robots.txt. See how a firewall can block it without telling you.
- Ten of 17 say nothing about AI crawlers. They rely on the default rules. That works, but naming the crawlers states your choice on purpose. See every AI crawler name and what to allow.
- Three sites returned 403, 418 and 202 to an ordinary request. From our machine they gave no robots.txt, no llms.txt and no home page. We cannot say whether real AI crawlers get the same, because bot protection often treats verified crawlers differently from a script. It does show that a plain visitor can hit a wall.
- One site is worth a second look. It returned 200 to a browser and 403 to both OpenAI crawler names. A request that only copies a crawler's name can be treated differently from the real crawler, so this is a flag to check in the server logs, not proof of a block.
- Just over half publish llms.txt. 12 of 20 served a real text file at /llms.txt. Whether any AI system uses it is not settled, so treat it as cheap and optional. See how to add llms.txt.
How we did it
- We took 20 sites that came up in searches on 13 November 2026 for AI search optimization advice, agencies and tools. It is a convenience sample of sites that teach the topic, not a random sample of the web.
- For each we requested /robots.txt, /llms.txt and the home page. A file counts as a robots.txt only if it returned 200, was not HTML and contained a User-agent line. llms.txt counts only if it returned 200 and was not an HTML page.
- We parsed the groups, and counted an AI crawler as blocked only when its own group contained "Disallow: /".
- For the home page we sent three requests: a Chrome browser name, and the published OAI-SearchBot and GPTBot names. These requests come from one ordinary IP address, not from OpenAI's, so sites that verify crawlers by IP may answer real crawlers differently.
What this does not show
- It is 20 sites, all in one niche. Do not read it as a rate for the web.
- It checks access, not whether AI systems cite anyone. Access is the first step, not the whole job.
- Results move. A firewall setting changed next week changes the answer.
Where devmarde.com stands
On the same day, devmarde.com named 7 AI crawlers in robots.txt, all allowed, served llms.txt, and answered 200 to all three request types. We ran the same script on ourselves. It is a starting point, not a result (how to tell if it is working).
Check your own site in a minute
Open yoursite.com/robots.txt and look for lines naming GPTBot or OAI-SearchBot with "Disallow: /". Then run the four quick tests here or run a Crawl Audit, which crawls as the AI bots do.
Questions people ask
Do most websites block ChatGPT?
Not in this sample. 0 of 17 readable robots.txt files blocked GPTBot, OAI-SearchBot or ChatGPT-User. Firewalls and bot protection can still block it without any line in robots.txt.
How many sites have an llms.txt file?
12 of the 20 sites we tested served one. All were in the AI search niche, so the share across all websites is likely much lower.
Can my site look fine to me and still block AI crawlers?
Yes. A firewall can show a normal page to you and a challenge page or a 403 to a crawler. That is why we test with crawler names and not only a browser.
Where can I see the raw data?
The per-site results and the script are kept on file. Write to hello@devmarde.com and we will send them. We do not name individual sites here because a copied crawler name cannot prove a real block.
Data collected 13 November 2026 by DevMarDe. Product and crawler names are trademarks of their owners.
Related
View all →AI crawler user agents: what to allow, from GPTBot to ClaudeBot
Every AI crawler name you will see in your logs, what each one is for, which to allow if you want to be found in AI answers, copy-paste robots.txt rules, and the test we ran on our own site.
Can ChatGPT read my website? Check it in 60 seconds
Four quick tests show whether ChatGPT, Claude and Perplexity can fetch and read your site: robots.txt, a crawler fetch, the raw HTML and your server logs. Free, no sign-up.
How to prioritize AI search fixes: what to do first
Not sure where to start with AI search? The order that works: let bots in, make the page readable, state who you are, then earn mentions. Fix the first one first.
Find out whether AI answers can read your site.
Crawl Audit scores your own domain against the signals in this guide. The free check needs no sign-up; the full report is $7.89, one time.