PerplexityBot in robots.txt: keep it allowed

Short answer: PerplexityBot is the crawler that puts websites into Perplexity's search answers, and Perplexity says it isn't used to train AI models. Blocking it means Perplexity can't read your pages to answer questions about you. For most businesses this is the one in this group to keep allowed. Crawl Audit check 3 confirms your robots.txt does.

What Crawl Audit checks

Crawl Audit fetches https://yourdomain.com/robots.txt and asks whether PerplexityBot may fetch your homepage (/). A User-agent: PerplexityBot group, matched in any letter case, takes priority over User-agent: *. Within the group the longest matching rule wins, and Allow wins a tie (RFC 9309). The check passes when PerplexityBot is allowed, or when there's no robots.txt. It fails when the group that applies disallows /. It also fails if robots.txt returns a server error (5xx) or can't be fetched at all, because under RFC 9309 crawlers then treat the whole site as blocked.

What PerplexityBot is

Perplexity's crawler documentation describes two user agents:

• PerplexityBot is "designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models."
• Perplexity-User fetches a page when a user's question needs it. Perplexity says: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."

So robots.txt is how you control PerplexityBot, the crawler that builds Perplexity's index. It is not a reliable way to stop Perplexity opening a page when someone asks for it. If a page must stay private, put it behind a login.

Why it's usually worth allowing

The other crawlers in Crawl Audit's checks 1 to 6, GPTBot, ClaudeBot, Google-Extended, Bytespider and CCBot, are about training. Blocking them is a legitimate opt-out. PerplexityBot is different: it's how your site gets into Perplexity's answers, so blocking it costs you visibility there. If check 3 fails and you didn't mean it to, the cause is usually one of these:

• A blanket block. User-agent: * with Disallow: / and no PerplexityBot group catches it.
• A copied "block AI bots" list that included PerplexityBot alongside the training crawlers.

A healthy setup that opts out of training but stays in Perplexity search:

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Allow: /

Perplexity says "it may take up to 24 hours for our systems to reflect changes" after you edit robots.txt.

robots.txt allows it, but a firewall may not

Check 3 only reads robots.txt. In August 2025 Cloudflare said of Perplexity that "we have de-listed them as a verified bot", so a Cloudflare rule that only lets verified bots through won't let PerplexityBot in. On Cloudflare, you need a rule that names it, checked against Perplexity's published IP list. Our guide to how firewalls block AI bots by accident covers the settings to look at.

How to change it on your platform

• WordPress: Yoast SEO › Tools › File editor or Rank Math › General Settings › Edit robots.txt. A physical robots.txt in the root overrides both.
• Shopify: in robots.txt.liquid, keep the robots.default_groups loop and add a User-agent: PerplexityBot / Allow: / group after it if a broader rule is catching it.
• Wix: SEO & GEO › Tools and settings › Robots.txt Editor.
• Squarespace: the Block known artificial intelligence crawlers setting doesn't include PerplexityBot. Squarespace's list covers training crawlers such as GPTBot, ClaudeBot and CCBot, so ticking it leaves Perplexity able to read the site.
• Next.js: in app/robots.ts, make sure no rule disallows / for PerplexityBot or *.

Is it really PerplexityBot?

Perplexity publishes its crawler IPs at perplexity.com/perplexitybot.json, and separately for Perplexity-User at perplexity-user.json. Compare visitors in your logs with those lists before trusting the name.

Sources

• Perplexity, Perplexity crawlers, checked 1 October 2026.
• Cloudflare blog, Perplexity and undeclared crawlers, 4 August 2025.
• IETF, RFC 9309: Robots Exclusion Protocol.
• Squarespace, Blocking AI crawlers; Shopify, Customize robots.txt.