ClaudeBot in robots.txt: should you block it?

Short answer: ClaudeBot is Anthropic's crawler for collecting web content that could be used to train its models. Blocking it opts your site out of that. It's a separate setting from Claude-SearchBot, which Anthropic uses for search results, and Claude-User, which fetches a page when someone asks Claude to. Crawl Audit check 2 shows whether your robots.txt lets ClaudeBot in.

What Crawl Audit checks

Crawl Audit reads https://yourdomain.com/robots.txt and decides whether ClaudeBot may fetch your homepage (/):

• A User-agent: ClaudeBot group, matched in any letter case, takes priority over User-agent: *.
• With no ClaudeBot group, the * rules apply.
• The longest matching rule wins, and Allow beats an equally long Disallow, per RFC 9309.

It passes when ClaudeBot may fetch /, or when there's no robots.txt. It fails when the applicable group disallows /, and the report quotes the rule that matched. It also fails if robots.txt returns a server error (5xx) or can't be fetched at all, because under RFC 9309 crawlers then treat the whole site as blocked.

What ClaudeBot is, and Anthropic's other bots

Anthropic's crawler page describes three user agents:

• ClaudeBot "helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training."
• Claude-SearchBot "navigates the web to improve search result quality for users."
• Claude-User: "When individuals ask questions to Claude, it may access websites using a Claude-User agent."

Anthropic says its bots honour robots.txt, and that applies to all three, Claude-User included. That differs from OpenAI's ChatGPT-User and Perplexity's Perplexity-User, whose operators say robots.txt may not apply.

Should you allow it?

As with any training crawler, it's your call. Blocking ClaudeBot opts out of training without touching Claude-SearchBot. If what you actually want is to stay readable in Claude's answers, check that you haven't blocked Claude-SearchBot or Claude-User by accident, for example with a * block or an "AI blocker" setting.

If ClaudeBot's crawling is the problem rather than the training, there's a middle option. Anthropic says it supports the non-standard Crawl-delay extension, so you can slow it down instead of shutting it out:

User-agent: ClaudeBot
Crawl-delay: 1

To opt out of training but stay in Claude's search and user fetches:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Subdomains need their own rules

robots.txt applies per host. Anthropic's page says the block has to be added "for every subdomain that you wish to opt out from." A rule on example.com doesn't cover shop.example.com or blog.example.com. Each serves its own /robots.txt. Crawl Audit checks the domain you entered.

How to change it on your platform

• WordPress: edit robots.txt in Yoast SEO › Tools › File editor or Rank Math › General Settings › Edit robots.txt. A physical robots.txt in the site root overrides both.
• Shopify: create robots.txt.liquid under Templates, keep the robots.default_groups loop, and add your ClaudeBot group after it.
• Wix: SEO & GEO › Tools and settings › Robots.txt Editor › View File.
• Squarespace: the Block known artificial intelligence crawlers setting under Settings › Crawlers includes ClaudeBot and anthropic-ai, but not Claude-SearchBot. There's no per-bot editor.
• Next.js: in app/robots.ts, add { userAgent: "ClaudeBot", disallow: "/" }, or crawlDelay: 1 for the slower option.

Common mistakes

• Blocking anthropic-ai and thinking that's ClaudeBot. Older guides list anthropic-ai. Anthropic's current page names ClaudeBot, Claude-SearchBot and Claude-User, so write rules for those.
• Forgetting that a named group replaces *. Once ClaudeBot has its own group, your * rules no longer apply to it.
• Assuming a 403 is robots.txt. A firewall can block ClaudeBot even when robots.txt allows it. See how firewalls block AI bots by accident.

Is it really ClaudeBot?

Anthropic publishes the IP addresses its crawlers use at claude.com/crawling/bots.json. Its page says: "If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic."

Sources

• Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 7 April 2026), checked 1 October 2026.
• IETF, RFC 9309: Robots Exclusion Protocol.
• Squarespace, Blocking AI crawlers; Shopify, Customize robots.txt; Next.js, robots.txt.