llms-txt
Publish llms.txt for documentation-heavy sites
Documentation portals often bury the most important pages behind dense navigation, search UIs, or framework-specific layouts. A curated llms.txt gives AI tools a cleaner entry point without replacing the normal crawl and indexing mechanisms of the web described by the llms.txt convention.
Quick Reference
- Serve
llms.txtat the root (/llms.txton the production domain) if you run a docs-heavy site - Curate a short list of stable, high-signal docs pages instead of dumping every URL
- Use
llms-full.txtonly as an optional companion when you can keep it current - Keep crawl and indexing controls in
robots.txt;llms.txtdoes not replace them
Check
Check whether this documentation-heavy site publishes llms.txt at the root and whether it contains a curated set of stable, high-value links such as getting-started guides, API references, concepts, and examples. Confirm that linked URLs return HTTP 200 and that llms.txt is not being treated as a replacement for robots.txt or XML sitemaps.
Fix
Add llms.txt at the site root with a short description of the project and a curated list of the most useful documentation URLs. If the docs set is large and you can maintain it, add an optional llms-full.txt companion and link to it from llms.txt. Keep crawl directives in robots.txt.