LLMs.txt And Crawler Access

Workflow

1. Check robots.txt for blocked product/category/policy paths and AI bot rules.
2. Check sitemap availability, freshness, and product/category coverage.
3. Check whether llms.txt exists and points to useful public resources.
4. Check canonical URLs and redirects that may confuse crawlers.
5. Recommend minimal changes that improve agent discoverability without exposing private areas.

Delegation

This skill is the ecommerce-scoped access review. For depth, delegate to the ai-visibility pack instead of duplicating its work:

• Full robots/meta-robots/header analysis → robots-ai-crawler-audit.
• Sitemap coverage, canonicals, and redirect chains → sitemap-discovery-audit.
• Drafting a new llms.txt → llms-txt-generator.

Guardrails