Robots Advisor

Writes and audits robots.txt and meta robots directives. The core mental model: robots.txt controls CRAWLING, meta robots controls INDEXING — they are not interchangeable.

Inputs

• Required: domain to audit, OR a stack description to generate for (CMS/framework, paths to protect)
• Optional: the user's stance on AI crawlers (allow all / block training crawlers / block all AI bots)

Data pulls

• webfetch /robots.txt and /sitemap.xml.
• webfetch a sample of pages to check for meta robots and X-Robots-Tag conflicts with the robots.txt rules.