Skip to content
DevMarDe
Crawl Audit · Next.js

Crawl Audit for Next.js sites

See what AI assistants see when they visit your site, and the exact code to fix it. Crawl Audit crawls your site the way GPTBot does and scores the signals that decide how easily an assistant can read and understand your business. A free run scores 25 of them. The paid report scores 200, ranked by impact, each with the exact code to paste. No call, no scoping document. SaaS products, marketing sites owned by engineering teams, and everything we build ourselves.

Open Crawl Audit About Crawl Audit
crawlaudit.devmarde.com/r/8f2aCrawl Audit · Findings
FindingImpact
GPTBot blocked by firewall403 from bot protection on every pageHigh
llms.txt missingNothing tells an AI crawler what the site is forHigh
Entity records unlinkedNo sameAs on Organization schemaMedium
31 pages without dateModifiedModels discount content they cannot dateMedium
$7.89
one time
6
checks
Live
status

What Next.js sites usually get wrong

  • The Metadata API covers titles and Open Graph but not JSON-LD; schema needs a script tag in the layout or page.
  • Vercel's bot protection is off by default; if Attack Challenge Mode is on, AI crawlers get a challenge page.
  • generateStaticParams makes programmatic pages cheap, which is why thin templated pages are the common failure.

Applying the result on Next.js

  • llms.txt on Next.js. Place llms.txt in /public. It deploys with the site and is served as text/plain.
  • Schema on Next.js. Render an @graph in app/layout.tsx for Organization and WebSite, and page-level blocks in each page component.
  • Crawlers on Next.js. Export a robots() function from app/robots.ts with explicit allow rules per AI user agent.

What Crawl Audit checks

  • Crawler accessrobots.txt, firewall rules and rate limits for four AI crawlers.
  • Schema depthOrganization, Person, Product, FAQ and Article validity.
  • Entity recordsWikidata, Crunchbase and sameAs coverage.
  • Answer structureWhether pages answer a question in the first paragraph.
  • FreshnessdateModified, changelogs and stale-content flags.
  • Citations todayWhere four assistants already mention you.
100
Lighthouse performance on this site. Same for SEO, accessibility and best practices
5
products of our own in production
3
practices, each with a named lead
1 day
reply time on every enquiry
4.9
from 9 reviews

Questions people ask

Does Crawl Audit work with Next.js?
Yes. It reads the public site, so the platform does not matter. The notes above cover where to apply the result on Next.js.
Is this a manual audit?
No. It is a tool. You enter a domain and it runs. Nobody reviews it by hand.
Why $7.89 and not free?
A real crawl costs us money. The price covers it and keeps the queue clear.
Do I need the other tools?
No. Crawl Audit tells you what is wrong; the free generators help you fix it.
What does the free preview on this site show?
A server-side fetch of your homepage, robots.txt, sitemap and llms.txt, scored against 25 of the signals. The full crawl runs at crawlaudit.devmarde.com.