CCBot in robots.txt: Common Crawl explained

Short answer: CCBot is the crawler run by Common Crawl, a non-profit that publishes an open archive of the web. That archive is, in Common Crawl's own words, "one of the most widely used sources of training data for large language models." Blocking CCBot keeps your future pages out of that dataset. It doesn't affect any AI search engine directly. Crawl Audit check 6 shows whether your robots.txt lets it in.

What Crawl Audit checks

Crawl Audit reads https://yourdomain.com/robots.txt and resolves the rules for CCBot on your homepage (/). A User-agent: CCBot group, in any letter case, takes priority over *, and the longest matching rule wins, with Allow winning a tie (RFC 9309). The check passes when CCBot may crawl, or when there's no robots.txt. It fails when the applicable rules disallow /. It also fails if robots.txt returns a server error (5xx) or can't be fetched at all, because under RFC 9309 crawlers then treat the whole site as blocked.

What CCBot is

Common Crawl's FAQ says CCBot identifies itself as CCBot/2.0 (https://commoncrawl.org/faq/) and is "an automated crawler, checking first the robots.txt, and if crawling a page is allowed, fetches pages". What makes it different from the other crawlers in Crawl Audit's checks 1 to 6 is what happens to the pages afterwards. They go into a public dataset anyone can download, and Common Crawl says that data, "typically after filtering and processing by third parties, has been a primary training data source for many large language models."

So a CCBot decision isn't about one AI company. Allowing it puts your public pages into a dataset that many model builders draw on. Blocking it keeps future crawls out of that dataset. Pages already in past archives stay there.

Should you allow it?

It's an open-data and training decision, and both answers are reasonable. Crawl Audit rates this check medium impact: if check 6 fails, confirm the block is intentional. To block it:

User-agent: CCBot
Disallow: /

Common Crawl adds: "We will periodically continue to check if the robots.txt file has been updated." If the concern is server load rather than the dataset, Common Crawl says "We obey the Crawl-delay parameter for robots.txt", so you can slow it down instead:

User-agent: CCBot
Crawl-delay: 2

How to change it on your platform

• WordPress: Yoast SEO › Tools › File editor or Rank Math › General Settings › Edit robots.txt. A physical robots.txt in the root overrides both.
• Shopify: add a User-agent: CCBot group in robots.txt.liquid, after the robots.default_groups loop.
• Wix: SEO & GEO › Tools and settings › Robots.txt Editor. Wix's own guide to blocking AI crawlers uses CCBot as one of its examples.
• Squarespace: Settings › Crawlers › Block known artificial intelligence crawlers includes CCBot.
• Next.js: { userAgent: "CCBot", disallow: "/" }, or crawlDelay: 2, in app/robots.ts.

Is it really CCBot?

Common Crawl warns that "we are aware of crawlers falsely identifying themselves as CCBot." The real one runs on dedicated IP ranges with reverse DNS. Hostnames look like 18-97-14-84.crawl.commoncrawl.org, except over IPv6, where reverse DNS isn't supported yet. Common Crawl publishes the ranges as JSON at index.commoncrawl.org/ccbot.json. If a "CCBot" in your logs ignores your robots.txt, check its IP against that list before blaming Common Crawl.

Common mistakes

• Expecting a block to remove existing data. robots.txt governs future crawls only.
• Blocking with *. User-agent: * / Disallow: / also stops search engines. Name CCBot instead.
• Forgetting subdomains. Each host serves its own robots.txt, so blog.example.com needs its own rule.

Sources

• Common Crawl, FAQ, CCBot and About, checked 1 October 2026.
• IETF, RFC 9309: Robots Exclusion Protocol.
• Wix, Blocking AI crawlers from your site; Squarespace, Blocking AI crawlers.