Web Scraping with mq

Three tools — pick the smallest one that fits:

• mq-crawl — fetch/crawl a URL to Markdown, with JS rendering, multi-page crawling, robots.txt handling.
• mq — filter/transform Markdown or HTML (-I html) with jq-like selectors. Full selector/function reference: processing-markdown skill.
• mq's http() builtin (--allow-net) — make the request inside a query, for cases mq-crawl can't (POST, custom headers/auth).

mq-crawl sends logs/stats to stderr — redirect with 2>/dev/null when scripting.

Fetch & Extract

mq-crawl --depth 0 https://example.com 2>/dev/null                       # page as Markdown (--depth 0 = don't follow links)
mq-crawl --depth 0 -q '.h | to_text()' https://example.com 2>/dev/null   # inline query, e.g. an outline

-q output is always Markdown/plain text, regardless of -f (which only formats the stats block on stderr, not content). For JSON/table/grep, skip -q and pipe the page into mq instead: