Google-Extended in robots.txt, explained

Short answer: Google-Extended isn't a crawler. It's a name you can use in robots.txt to tell Google whether your content may be used to train Gemini models and to ground Gemini answers. Google says it doesn't affect whether you appear in Google Search, and it isn't a ranking signal. Blocking it doesn't remove you from AI Overviews either. Crawl Audit check 4 shows what your robots.txt currently says to it.

What Crawl Audit checks

Crawl Audit reads https://yourdomain.com/robots.txt and resolves the rules for Google-Extended on your homepage (/), the same way the other crawler checks do. A User-agent: Google-Extended group, in any letter case, takes priority over *, and the longest matching rule wins. The check passes when Google-Extended is allowed, or when there's no robots.txt. It fails when the applicable rules disallow /. It also fails if robots.txt returns a server error (5xx) or can't be fetched at all, because under RFC 9309 crawlers then treat the whole site as blocked.

What Google-Extended controls

Google's crawler documentation says Google-Extended lets you manage whether content Google crawls "may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding". It also makes two points that matter here:

• There's no Google-Extended crawler. "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings". Your pages are fetched by Google's normal crawlers, and the token only says what the content may then be used for.
• Search is unaffected. "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

What controls AI Overviews, then?

Google's guidance on AI features in Search says "robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search". To limit what's shown from your pages, including in AI features, use nosnippet, data-nosnippet, max-snippet or noindex. In practice:

• Blocking Google-Extended: opts out of Gemini training and grounding. No effect on Search or AI Overviews.
• Blocking Googlebot: removes you from Google Search, including AI Overviews. Almost never what a business wants.
• Using nosnippet or max-snippet: limits how much text Google shows from a page, without leaving Search.

Should you allow it?

It's a training and grounding decision, not a search one. If you're content for Gemini to learn from and draw on your public pages, leave it allowed, which is the default. If you'd rather opt out, block it, and expect no change in Google Search:

User-agent: Google-Extended
Disallow: /

You can't test it by fetching as Google-Extended

Because Google never sends a request that calls itself Google-Extended, a test that fetches your page with "Google-Extended" as the user agent only tells you whether your firewall happens to match that string. It says nothing about what Google does. Reading robots.txt, which is what check 4 does, is the meaningful test.

How to change it on your platform

• WordPress: Yoast SEO › Tools › File editor or Rank Math › General Settings › Edit robots.txt.
• Shopify: add a User-agent: Google-Extended group in robots.txt.liquid, after the default-rules loop.
• Wix: SEO & GEO › Tools and settings › Robots.txt Editor.
• Squarespace: the Block known artificial intelligence crawlers setting includes Google-Extended. It also includes GoogleOther, a separate Google user agent for one-off crawls, which Google says has no effect on Search.
• Next.js: { userAgent: "Google-Extended", disallow: "/" } in app/robots.ts.

Common mistakes

• Blocking Googlebot to "opt out of AI". That takes you out of Google Search entirely. Use Google-Extended for Gemini, and snippet controls for how much Search shows.
• Expecting the change to hide you from AI Overviews. It won't. Google-Extended doesn't control them.

Sources

• Google, Google's common crawlers (updated 14 July 2026), checked 1 October 2026.
• Google Search Central, AI features and your website (updated 10 December 2025).
• IETF, RFC 9309: Robots Exclusion Protocol.
• Squarespace, Blocking AI crawlers.