Meta robots noindex tag: Crawl Audit check 39

Short answer: noindex tells search engines not to show a page in their results. Check 39 fails when the audited page has a noindex in its robots meta tag, and for a homepage or service page that's almost always a mistake, often a setting left on from when the site was being built. On pages you really want out of search, such as a thank-you page or a client login, noindex is the right tool.

What Crawl Audit checks

• Fail: <meta name="robots"> contains noindex.
• Pass: it doesn't.

What it doesn't see: a noindex sent as an HTTP header (X-Robots-Tag), or in a crawler-specific tag like <meta name="googlebot">. If a page won't index and this check passes, look at those too.

What Google says

• noindex: "Do not show this page, media, or resource in search results."
• The header works too: "Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag."
• The page has to be crawlable: "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file". A page blocked in robots.txt can't be read, so Google never sees its noindex.
• "Specifying the noindex rule in the robots.txt file is not supported by Google."

Bing works the same way: "we need to be able to crawl the page to see this tag, so do not block access to the page."

On AI crawlers: OpenAI, Anthropic and Perplexity document how their bots treat robots.txt, but their crawler pages don't say whether they honour noindex. Don't rely on it to keep a page out of AI answers.

The settings that add noindex by accident

• WordPress: Settings › Reading › Discourage search engines from indexing this site. Since WordPress 5.3 it adds a robots meta tag with noindex,nofollow to every page. It's often ticked while a site is being built and forgotten at launch. Yoast also has a per-post setting: "Allow search engines to show this content in search results?" set to No.
• Wix: each page has "Let search engines index this page". Hide adds <meta name="robots" content="noindex">. There's also a site-wide "Let search engines index your site" toggle.
• Squarespace: "Hide page from search results" adds noindex and removes the page from the sitemap. Careful with the site-wide option, "Block Search engine crawlers": it adds Disallow: / to robots.txt instead of noindex, so under Google's rule above, Google can't even see a noindex on those pages.
• Shopify: setting the seo.hidden metafield to 1 hides a page, post or product "from sitemaps, search engines, and your online store search."
• Next.js: a robots: { index: false } left in shared metadata applies to every page that inherits it.

Our pages send index, follow, and only the admin area and the API are kept out, through robots.txt.

Sources

• Google Search Central, Robots meta tag and X-Robots-Tag specifications (updated 24 March 2026) and Block search indexing with noindex, checked 3 October 2026.
• Bing, Robots meta tags Bing supports.
• WordPress, Settings Reading screen; Yoast, How to noindex URLs; Wix, Default SEO settings; Squarespace, Hiding pages from search; Shopify, Hide a page from search engines.