robots-meta-conflict

Installation
SKILL.md

Robots Meta Conflict

Pages that appear in search results despite your intention to deindex them are usually blocked in robots.txt — Google cannot read the noindex directive it cannot fetch.

Quick Reference

  • If a page is blocked in robots.txt, crawlers never fetch it and therefore never read its noindex meta tag
  • To properly deindex a page, use <meta name="robots" content="noindex"> without blocking it in robots.txt
  • If you block a URL in robots.txt, Google may still show it in results using anchor text from backlinks
  • Audit pages that need to be deindexed by confirming they are crawlable but carry noindex

Check

Cross-reference the site's robots.txt disallow paths against pages carrying <meta name="robots" content="noindex">. Flag any page where the URL matches a Disallow rule AND has a noindex tag — the noindex will never be seen by crawlers.

Fix

For pages that must be deindexed: remove the robots.txt Disallow rule for that URL and keep the noindex meta tag. For pages that must simply be uncrawled without appearing: keep the Disallow rule and remove the noindex tag (understanding Google may still show the URL).

Installs
1
GitHub Stars
73.0K
First Seen
Jun 7, 2026
robots-meta-conflict — thedaviddias/frontendchecklist