NERDTOOLS / DOMAIN DIAGNOSTICS

NerdTools guide

How to find and fix a noindex directive

A noindex directive tells supporting search engines not to include a page in search results. It may appear in HTML meta robots or the HTTP X-Robots-Tag header.

Reviewed by NerdTools · Diagnostic guidance · Methods and limitations

Inspect the final response

Follow redirects and inspect both response headers and rendered HTML on the destination. Check crawler-specific directives as well as the general robots instruction.

Ensure the crawler is allowed to fetch the page; if robots.txt blocks it, the crawler may not see a noindex change promptly.

Remove every conflicting source

Update the CMS, template, plugin, reverse proxy, or CDN rule that emits noindex. Confirm the canonical points to an indexable equivalent and that internal links and the sitemap use the intended URL.

Request recrawling only after the live final response consistently shows the corrected directives.

Sources and standards

FAQ

Can canonical override noindex?

No. Do not combine noindex with the expectation that canonical signals will transfer normally.

Can X-Robots-Tag affect non-HTML files?

Yes. HTTP headers can control indexing for PDFs and other supported resources without HTML metadata.

How quickly is removal or reindexing processed?

Timing varies with crawl demand and site signals. Use Search Console inspection for Google-specific status.

Related diagnostics