robots.txt presence and syntax: Crawl Audit check 31

Short answer: robots.txt is a plain text file at the root of your site that tells crawlers which URLs they may visit. Check 31 fails if there's no robots.txt and warns if the file has lines Crawl Audit can't parse. To be clear: Google doesn't require one. Its rule is that a missing file means "no crawl restrictions". A robots.txt is still worth having, because it's where you point crawlers to your sitemap and state your rules for AI crawlers on purpose.

What Crawl Audit checks

• Pass: /robots.txt returns 200 and every line parses.
• Warning: it returns 200, but some lines are invalid: not in field: value form, an unknown directive, or an Allow or Disallow before any User-agent. The report quotes up to three of them.
• Fail: it doesn't return 200.

A robots.txt that returns a server error (5xx) is more serious than a missing one: checks 1 to 6 fail too, because under the standard, crawlers then treat the whole site as blocked.

What Google and the standard say

• Not required: "If you're a small site, you probably don't need a robots.txt file."
• A missing file: "Google's crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn't exist. This means that Google assumes that there are no crawl restrictions."
• Invalid lines: "Google ignores invalid lines in robots.txt files ... and use only valid lines." One bad line won't break the file for Google, but other crawlers may be less forgiving, which is why Crawl Audit flags it.
• Size: "Google enforces a robots.txt file size limit of 500 kibibytes (KiB). Content which is after the maximum file size is ignored."
• Location: the file must be "in the top-level directory of a site", and its rules "apply only to the host, protocol, and port number where the robots.txt file is hosted." A subdomain needs its own.
• Not a privacy tool: robots.txt "is not a mechanism for keeping a web page out of Google." To keep a page out of search, use noindex (check 39).

A sensible minimal file

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Ours is longer, at 2,068 bytes, because it names 20 user agents, mostly AI crawlers, one by one. It's still nowhere near the 500 KiB limit.

On your platform

• WordPress: WordPress serves a virtual robots.txt, and since version 5.5 it points to the built-in sitemap. Edit it with Yoast (Tools › File editor) or Rank Math (General Settings › Edit robots.txt).
• Shopify: every store has a default robots.txt. Customise it with a robots.txt.liquid template.
• Wix: SEO & GEO › Tools and settings › Robots.txt Editor.
• Squarespace: you can't edit it directly. Squarespace manages the file for you.
• Next.js: app/robots.ts, or a route that returns plain text if you need comments.

Sources

• Google, How Google interprets the robots.txt specification (updated 31 August 2026) and Introduction to robots.txt, checked 3 October 2026.
• Google Search Console Help, robots.txt report.
• IETF, RFC 9309: Robots Exclusion Protocol.