Robots AI Crawler Audit

Check whether crawler access rules help or block AI visibility.

Scope: access rules only. Sitemap coverage belongs to sitemap-discovery-audit; drafting llms.txt belongs to llms-txt-generator; whole-site triage belongs to ai-visibility-audit.

Workflow

1. Fetch /robots.txt.
2. Identify global disallow rules, sitemap declarations, crawl delays, and user-agent specific rules. On ecommerce/marketplace sites, check specifically whether product, category, and policy paths are blocked — these are usually the pages a shopper or AI agent needs to read.
3. Check page-level noindex, nofollow, canonical tags, and X-Robots-Tag headers on key URLs.
4. Check response security headers (Strict-Transport-Security, X-Content-Type-Options, X-Frame-Options) on key URLs — a real technical-SEO/trust signal, and missing ones can also indicate a misconfigured origin worth flagging alongside crawler-access findings.
5. Look for AI crawler-specific rules for major bots where visible.
6. Compare access rules against the user's visibility goals.
7. [EXPERIMENTAL] Check for emerging draft signals (Content Signals directives in robots.txt, Web Bot Auth endpoints/headers, and DNS-AID TXT records) — see references/checks.md. Clearly mark any findings in this category as experimental draft standards that do not block core search/crawler indexing.
8. Recommend exact changes only when the desired access policy is clear.

Output