Technical
Crawler Tester
Test whether GPTBot, PerplexityBot, ClaudeBot and Google-Extended can actually fetch your pages. Most blocked sites do not know they are blocked.
Free10 second test
- Four crawler user agents
- robots.txt rule per agent
- Status code per fetch
- Fix suggestions
We fetch the URL four times, once with each AI crawler's user agent, and read your robots.txt for each. The result shows the status code, the rule that applied and what to change.
Public URLs only. A handful of requests; nothing is stored.
Real requests
We fetch with each crawler user agent and report the status code.
robots.txt, parsed
Shows which rule applies to each agent, not just whether the file exists.
Per-page detail
Homepage passing does not mean your blog is reachable. Test any URL.
Four crawlers, any URL
- 01GPTBotOpenAI crawler access.
- 02PerplexityBotPerplexity crawler access.
- 03ClaudeBotAnthropic crawler access.
- 04Google-ExtendedGemini grounding access.
- 05robots.txtDirectives that apply to each agent.
- 06Edge rules403 and 429 responses that robots.txt does not explain.
Step-by-step articles that use this tool.
Questions people ask
- Should I allow every crawler?
- Not necessarily. This tells you what is happening so the choice is deliberate.
- Does it slow my site?
- No. It is a handful of requests.
- Can I test staging?
- Only if it is publicly reachable.
Would rather we did it?
The AI visibility service starts from your audit and clears the high-impact findings in two to four weeks.