GPTBot in robots.txt: should you block it?

Short answer: GPTBot is OpenAI's crawler for collecting pages that may be used to train its models. Blocking it in robots.txt opts your content out of that training. It does not take you out of ChatGPT's search answers, which use a different crawler, OAI-SearchBot. Crawl Audit check 1 tells you which of the two outcomes your robots.txt produces for GPTBot, so the choice is deliberate.

What Crawl Audit checks

Crawl Audit fetches https://yourdomain.com/robots.txt and works out what GPTBot is allowed to do on your homepage (/), following the robots.txt standard, RFC 9309:

• If there is a group for User-agent: GPTBot, GPTBot obeys that group and ignores User-agent: *. Names are matched without regard to case.
• If there is no GPTBot group, it obeys the * group.
• Within the group, the longest matching rule wins. If an Allow and a Disallow are equally long, Allow wins.

The check passes when GPTBot may fetch /, or when the site has no robots.txt (a missing file means everything is open). It fails when the rules that apply to GPTBot disallow /, and the report shows the exact rule that matched. It also fails if robots.txt returns a server error (5xx) or can't be fetched at all, because under RFC 9309 crawlers then treat the whole site as blocked.

What GPTBot is

OpenAI runs three separate user agents, and its documentation says each robots.txt setting "is independent of the others":

• GPTBot is used "to crawl content that may be used in training our generative AI foundation models."
• OAI-SearchBot is used "to surface websites in search results in ChatGPT's search features."
• ChatGPT-User fetches a page when someone asks ChatGPT to. OpenAI says that because these visits are started by a user, "robots.txt rules may not apply."

So you can allow OAI-SearchBot, to stay in ChatGPT search, and disallow GPTBot, to opt out of training. OpenAI also notes that if you allow both, "we may use the results from just one crawl for both use cases."

Should you allow it?

That's a business decision, and both answers are legitimate. Allowing GPTBot lets your public content be considered for future model training. Blocking it opts out of that, at no cost to ChatGPT search. Which means a failing check 1 isn't automatically a problem. It's a prompt to confirm the block is on purpose, and that it hasn't also caught OAI-SearchBot.

The mistake we see is a blanket User-agent: * / Disallow: / left over from a staging site, or an "AI blocker" plugin that adds every AI user agent. Either can take the search bots out along with GPTBot. The free AI visibility check on this site tests the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) for exactly this reason.

Check it yourself

Open https://yourdomain.com/robots.txt and look for a User-agent: GPTBot group. If there isn't one, read the User-agent: * group instead. To opt out of training but stay in ChatGPT search:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

To allow both, you don't need a GPTBot group at all, as long as the * group doesn't disallow /. OpenAI says it can take about 24 hours after a robots.txt change for its systems to adjust.

How to change it on your platform

• WordPress: WordPress serves a virtual robots.txt unless a real file sits in your site's root. In Yoast SEO, go to Yoast SEO › Tools › File editor. In Rank Math, go to General Settings › Edit robots.txt (Advanced Mode). If a physical robots.txt exists in the root, edit that file instead, because it overrides both plugins.
• Shopify: add a robots.txt.liquid template under Templates in the theme code editor. Keep Shopify's loop over robots.default_groups, then add your User-agent: GPTBot group after it.
• Wix: open SEO & GEO › Tools and settings › Robots.txt Editor › View File, add the group and save. There's a Reset to Default button if you need to undo it.
• Squarespace: there's no robots.txt editor. The Settings › Crawlers › Block known artificial intelligence crawlers box blocks GPTBot, along with a list of other AI user agents.
• Next.js: add a rule to app/robots.ts, e.g. { userAgent: "GPTBot", disallow: "/" } in the rules array.

Common mistakes

• A GPTBot group replaces *, it doesn't add to it. If you give GPTBot its own group, rules you wrote under *, such as Disallow: /admin/, no longer apply to it. Repeat them in the GPTBot group if you want them.
• An empty Disallow: means "allow everything". It isn't a block.
• robots.txt isn't a lock. It tells well-behaved crawlers what you want, but a firewall or CDN can still block a bot your robots.txt allows. If robots.txt says yes and you still see 403s, read how firewalls block AI bots by accident.
• A robots.txt that errors blocks everything. Under RFC 9309, if robots.txt returns a server error (5xx), crawlers "MUST assume complete disallow". Make sure the file returns 200, or 404 if you have no rules.

Is it really GPTBot?

Anyone can send a request that calls itself GPTBot. OpenAI publishes the IP ranges its crawler uses at openai.com/gptbot.json, so you can check a visitor in your server logs against that list before trusting the name.

Sources

• OpenAI, Overview of OpenAI crawlers, checked 1 October 2026.
• IETF, RFC 9309: Robots Exclusion Protocol.
• Yoast, How to edit robots.txt through Yoast SEO; Rank Math, How to edit robots.txt with Rank Math.
• Shopify, Customize robots.txt; Wix, Editing your site's robots.txt file; Squarespace, Blocking AI crawlers; Next.js, robots.txt.