Docs Fetch Ladder

When to use

Use when the user asks to fetch documentation, read a vendor's docs, gather API references, or scrape a documentation site. Keep the read bounded to the question and the user's authorized URLs.

Steps

1. Identify the question, base domain, pages needed, and available fetch tools. For one answer, prefer specific pages over a full-site crawl.
2. Fetch <origin>/llms.txt and <origin>/docs/llms.txt together when independent calls can run concurrently. Check the response status and content. If either is a useful index, select the links that answer the question. An index is a navigation aid, not proof that every page or claim is current.
3. Fetch the relevant Markdown pages. Prefer .md links explicitly listed in the index. If only a page URL is known, try its .md variant once where plausible. Preserve relevant query parameters and resolve relative links against the response URL. Batch independent page reads.
4. If Markdown is unavailable, empty, or missing the needed detail, fetch the specific plain page with the available HTTP or page-reading tool. Check that the body contains the requested documentation rather than a login, error, or navigation shell.
5. If a plain fetch cannot supply the needed content, use an available crawler. Crawl4AI is one option: crwl <url> -o markdown for a page, or crwl <url> -o markdown-fit for documentation or an article. For a site crawl, set a page bound appropriate to the request. Check the installed tool's help before using options beyond these examples.
6. If the crawler fails, returns no useful content, or lacks a required capability, use a hosted scraper as the last resort within the user's permissions and budget. State the failed rung and reason. If no permitted tool can fetch the content, report the limit and the specific missing source.
7. Stop escalating once the requested content is usable. Cite the content page, name the rung that supplied it, and note gaps that affect the answer.

If the index is missing, continue to known pages at step 3. If no page URL is known, use a bounded plain-page read to discover relevant links before escalating. If multiple pages require different rungs, report them separately.

Output shape