Crawl audit checklist, including the AI crawlers

Short answer: a crawl audit checks whether search engines and AI crawlers can reach, fetch and read every page that matters on your site, and whether they waste their visits on things that aren't pages. It used to have one audience, Googlebot. Now it has two: the search bots, and the AI crawlers behind ChatGPT, Claude and Perplexity, which behave differently and fail in different places. This is the checklist I use, with the numbers from our own audit.

What a crawl audit actually checks

Four questions, in this order. Can a crawler reach the page (robots.txt, firewall, links)? Can it fetch it (status codes, redirects, speed)? Can it read it (the text is in the HTML, not added later by JavaScript)? And is it spending its visits on the right pages, rather than on duplicates, parameters and files that aren't pages at all?

A technical SEO audit covers more ground: titles, structured data, Core Web Vitals, content. A crawl audit is the narrower first step. If it fails, nothing else on the list matters, because the page never gets read.

Our own crawl audit, with the real numbers

I ran this on devmarde.com in October, about three weeks after our Search Console data began. Every number below is from our own Google Search Console.

So the site was healthy in the usual sense. Fast, almost no errors. And it was still failing the crawl audit, because Google spent four out of five visits on data files and the pages we cared about weren't even known to it.

What we changed on 7 October: noindex, follow on the 448 templated pages, a sitemap cut from 506 URLs to 59, and one robots.txt line, Disallow: /*?_rsc=. Google's robots.txt rules support the * wildcard, so one line covers every data file. The first load of every page is still plain HTML, so nothing a crawler needs is hidden. I'll add the after numbers here once there are 28 days of crawl data to compare, not before.

The classic checklist

1. robots.txt is reachable, valid and deliberate

It should return 200 at /robots.txt, parse cleanly and name your sitemap. Every Disallow should have a reason you can say out loud. A blanket Disallow: / under User-agent: * is the most expensive mistake on this list. Crawl Audit checks: robots.txt syntax, sitemap reference in robots.txt.

2. The sitemap lists only pages you want indexed

Every URL in it should return 200, be canonical and be indexable. A sitemap full of redirects, 404s or noindex pages teaches Google not to trust it. Ours listed 448 pages we didn't need indexed, which is why it went from 506 URLs to 59. Checks: sitemap accessibility, sitemap size.

3. Status codes and redirects are clean

Important pages answer 200. Moved pages use one permanent redirect (301 or 308), not a chain and not a temporary 302 left over from a migration. HTTP goes to HTTPS in one hop. Checks: response code, redirect chains, 302s, HTTP to HTTPS.

4. Canonicals point where you mean

Every indexable page has a self-referencing canonical with the full URL, on the same host and protocol you serve. Pick www or not, a trailing slash or not, and redirect the other version. Checks: self-referencing canonical, trailing slash consistency.

5. Every page that matters is linked

Crawlers find pages by following links. A page that only exists in the sitemap, with no internal links, is an orphan, and orphans get crawled last. Checks: orphan pages, broken internal links.

6. Crawl budget isn't spent on things that aren't pages

This is the one most audits miss, and the one that hit us. Look at Settings > Crawl stats in Search Console and open the breakdown by file type. If HTML is a small share, click the biggest type and read the example URLs. Common culprits: URL parameters, faceted navigation, internal search results, and framework data files like Next.js ?_rsc=. Google's own guidance is to block URLs you don't want crawled and keep the sitemap to pages you do.

7. The server answers fast and the same way every time

Slow or erratic responses make Google crawl less. A 5xx rate above zero in the crawl stats is worth chasing. Check: time to first byte.

8. noindex is only where you put it on purpose

A leftover noindex from a staging site, or a CMS setting ticked during a launch, quietly removes pages from Google. Check every important template. Check: meta noindex.

The AI-crawler half most crawl audits skip

Everything above is about Googlebot. AI crawlers need the same basics and fail in three extra places.

9. Each AI crawler is named in robots.txt, on purpose

There isn't one "ChatGPT bot". OpenAI, Anthropic and Perplexity each run separate crawlers for search, for training, and for fetching a page when a user asks. The search ones (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are the ones that decide whether you can show up in their answers. Decide on each one and write it down. The full table is in Can ChatGPT read my website?. Checks: GPTBot, ClaudeBot, PerplexityBot, Google-Extended.

10. The firewall lets them in

robots.txt can say yes while your CDN or firewall says no. Cloudflare's AI controls, bot-fight modes, challenge pages and WAF rules all block AI crawlers by accident, and robots.txt never sees it. Fetch a page with each crawler's name and look for a 403 or a challenge. How to read the result and fix it: Is your firewall blocking ChatGPT?

11. The words are in the HTML

Vercel's study of AI crawler traffic found that none of the major AI crawlers run JavaScript. They download your script files and never execute them. If your text only appears after scripts run, Googlebot may still see it, because Google renders pages, but the AI crawlers get an empty page. Open view-source and search for a sentence you can see on screen. Checks: render-blocking JavaScript, hero rendering delay.

12. Give them a map: llms.txt and structured data

An llms.txt file lists your most important pages for language models, and Organization and product structured data tells them who you are in a form they can read. Neither replaces the basics, but both are cheap. Generate one with the free llms.txt generator. Check: Organization schema.

Run it yourself in five minutes

1. Open yourdomain.com/robots.txt and yourdomain.com/sitemap.xml. Both should load.
2. In Search Console, open Settings > Crawl stats and look at "By file type" and "By response".
3. Fetch your homepage as an AI crawler:

   curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot/1.0; +https://openai.com/searchbot" https://yourdomain.com/

   Then repeat with PerplexityBot and Claude-SearchBot.
4. Press Ctrl+U on your homepage and search for your opening line.
5. Inspect your three most important URLs in Search Console's URL Inspection. "URL is unknown to Google" on a money page is a crawl problem, not a content problem.

Or run the free Crawl Audit check. It fetches your homepage, robots.txt, sitemap and llms.txt and runs 25 of the 200 checks, including the robots.txt rules for the AI crawlers, with a fix for each one that fails. The full report runs all 200 for $7.89, one time.

How often to run a crawl audit

After every redesign, platform move or CDN and firewall change, because those are when crawl problems appear without anyone noticing. Otherwise once a quarter. And look at the crawl stats file-type split once a month. It takes a minute, and it's the number that told us something was wrong.

Questions people ask

What is a crawl audit?

A check that search engines and AI crawlers can reach, fetch and read the pages you want found, and that they aren't spending their visits on pages and files you don't need crawled. It covers robots.txt, sitemaps, status codes, redirects, canonicals, internal links, crawl budget and, now, AI crawler access.

Is a crawl audit the same as a technical SEO audit?

No. A crawl audit is the first part of a technical SEO audit. It answers whether pages can be crawled and read at all. A technical audit then looks at how well they're built: titles, structured data, speed, content.

Do AI crawlers follow robots.txt?

The search and training crawlers from OpenAI, Anthropic and Perplexity say they do. The fetchers that act for a user, such as ChatGPT-User and Perplexity-User, may not, because a person asked for the page. And a firewall can block a crawler that robots.txt allows.

Why is Google crawling files that aren't pages?

Because it finds them in links and in what your pages load. Modern frameworks fetch data files when a visitor clicks around, and Googlebot follows the same paths. Search Console's crawl stats show it under "Other file type". If those URLs aren't pages, block them in robots.txt with a wildcard rule.

How long does a crawl audit take?

The five-minute version above finds most problems. A full one on a site with a few hundred pages takes a few hours, most of it reading Search Console reports and fixing what they show.

Sources

• Google, Crawl Stats report, Search Console Help, checked 8 October 2026.
• Google, How Google interprets the robots.txt specification (wildcards), checked 8 October 2026.
• Google, Crawl budget management, checked 8 October 2026.
• OpenAI, Overview of OpenAI crawlers; Anthropic, Anthropic's web crawling bots; Perplexity, Perplexity crawlers, checked 30 September 2026.
• Vercel, The rise of the AI crawler, 17 December 2024.
• devmarde.com's own Google Search Console: Crawl stats (17 September to 6 October 2026) and URL Inspection (7 October 2026).

Zain Kazmi runs DevMarDe, a development, marketing and design studio. We built Crawl Audit and run it on our own site first.