Avoid conflicting indexability signals

Conflicting indexability directives create unpredictable crawling and indexing behaviour. The most dangerous combination is robots.txt blocking a page that also has a noindex tag — the noindex is never read, but the URL is still known to Google, leaving it in a limbo state that wastes crawl budget.

Quick Reference

• robots.txt blocks crawling; noindex blocks indexing — they are different mechanisms and should not be applied together
• A page blocked in robots.txt cannot receive a noindex directive because crawlers never read the page
• Canonical tags pointing to a noindex page create an unresolvable conflict — canonicalise to an indexable URL instead

Check

For each page, collect four signals: (1) Is the URL path blocked by robots.txt? (2) Does the page HTML contain <meta name='robots' content='noindex'>? (3) Does the HTTP response include an X-Robots-Tag: noindex header? (4) Does the page's <link rel='canonical'> point to a different URL? Flag: pages blocked in robots.txt that also have noindex directives, pages with canonical pointing to a noindex URL, and pages with conflicting index/noindex signals from meta and HTTP header.

Fix