Bytespider in robots.txt: what's known, check 5

Short answer: Bytespider is a crawler operated by ByteDance, the company that owns TikTok. Cloudflare says it is "reportedly used to gather training data" for ByteDance's language models. ByteDance publishes no crawler documentation we could find, so its exact purpose and its robots.txt behaviour aren't officially stated. Crawl Audit check 5 shows whether your robots.txt allows it. It's rated low impact, because blocking a training crawler is a legitimate choice.

What Crawl Audit checks

Crawl Audit reads https://yourdomain.com/robots.txt and works out whether Bytespider may fetch your homepage (/). A User-agent: Bytespider group, in any letter case, takes priority over *, and the longest matching rule wins (RFC 9309). The check passes when Bytespider is allowed, or when there's no robots.txt. It fails when the rules that apply disallow /. It also fails if robots.txt returns a server error (5xx) or can't be fetched at all, because under RFC 9309 crawlers then treat the whole site as blocked.

What's known about Bytespider

We looked for a first-party source, the way OpenAI, Anthropic and Perplexity document their crawlers. bytedance.com/en/bytespider returns a "not found" page, and we found no other official page describing the bot. What follows is therefore third-party, and we've labelled it as such:

• Cloudflare, July 2024: Bytespider is "reportedly used to gather training data for its large language models (LLMs)". It accessed 40.40% of the websites in Cloudflare's sample, and "not only leads in terms of number of requests but also in both the extent of its Internet property crawling and the frequency with which it is blocked."
• Cloudflare, July 2025: Bytespider's share of AI crawler traffic fell from 42% in May 2024 to 7.2% in May 2025, and "Bytespider plummeted 85% in request volume".
• Cloudflare's AI Crawl Control reference lists Bytespider with operator "ByteDance" and category "AI Crawler".

Without a first-party statement, we can't tell you whether Bytespider honours robots.txt today. You'll see claims either way on SEO blogs, but none from ByteDance.

Should you allow it?

Bytespider isn't the crawler behind ChatGPT, Claude, Perplexity or Google's answers, and it's described as collecting training data. So, like GPTBot or CCBot, allowing or blocking it is a business decision. That's why Crawl Audit rates this check low impact: a failing check 5 on its own is nothing to worry about if the block is deliberate.

To block it in robots.txt:

User-agent: Bytespider
Disallow: /

Because its robots.txt compliance isn't documented, robots.txt alone may not be enough if you want it gone. Check your server logs after the change. If it keeps crawling, your CDN or firewall can block it by user agent. Cloudflare's AI crawler controls cover it, and our guide to how firewalls handle AI bots explains those settings.

How to change it on your platform

• WordPress: Yoast SEO › Tools › File editor or Rank Math › General Settings › Edit robots.txt.
• Shopify: add a User-agent: Bytespider group in robots.txt.liquid, after the default-rules loop.
• Wix: SEO & GEO › Tools and settings › Robots.txt Editor.
• Squarespace: Settings › Crawlers › Block known artificial intelligence crawlers includes Bytespider, and TikTokSpider too.
• Next.js: { userAgent: "Bytespider", disallow: "/" } in app/robots.ts.

Common mistakes

• Treating a failing check 5 as urgent. It's low impact. Fix the search-crawler checks first, such as PerplexityBot (check 3).
• Blocking it with *. A User-agent: * / Disallow: / to stop Bytespider stops every other crawler without its own group too, including search engines.

Sources

• Cloudflare blog, Declaring your AIndependence, 3 July 2024 (third-party data on Bytespider).
• Cloudflare blog, From Googlebot to GPTBot: who's crawling your site in 2025, 1 July 2025.
• Cloudflare, AI Crawl Control bot reference, checked 1 October 2026.
• IETF, RFC 9309: Robots Exclusion Protocol; Squarespace, Blocking AI crawlers.