Crawler Tester for WordPress sites
Test whether GPTBot, PerplexityBot, ClaudeBot and Google-Extended can actually fetch your pages. Most blocked sites do not know they are blocked. Most SMB marketing sites and a large share of clinic, school and professional-services sites.
- Free
- 10 second test
- 6
- checks
- Live
- status
What WordPress sites usually get wrong
- Security plugins such as Wordfence and hosts such as WP Engine often block unknown user agents by default; GPTBot returns 403 and nobody notices.
- Yoast and Rank Math emit Organization schema but rarely include sameAs or a stable @id.
- dateModified is emitted, but only updates when the post is saved, so bulk edits inflate freshness.
Applying the result on WordPress
- How to change crawler rules on WordPress. Settings → Reading controls only the noindex flag. Edit robots.txt directly or through Yoast's file editor to allow GPTBot, PerplexityBot, ClaudeBot and Google-Extended.
What Crawler Tester checks
- GPTBotOpenAI crawler access.
- PerplexityBotPerplexity crawler access.
- ClaudeBotAnthropic crawler access.
- Google-ExtendedGemini grounding access.
- robots.txtDirectives that apply to each agent.
- Edge rules403 and 429 responses that robots.txt does not explain.
100
Lighthouse performance on this site. Same for SEO, accessibility and best practices
5
products of our own in production
3
practices, each with a named lead
1 day
reply time on every enquiry
4.9
from 9 reviews
Questions people ask
- Does Crawler Tester work with WordPress?
- Yes. It reads the public site, so the platform does not matter. The notes above cover where to apply the result on WordPress.
- Should I allow every crawler?
- Not necessarily. This tells you what is happening so the choice is deliberate.
- Does it slow my site?
- No. It is a handful of requests.
- Can I test staging?
- Only if it is publicly reachable.