markdown.new-crawl

Markdown.New Crawl Local-First Access

Use markdown.new/crawl for multi-page crawling and async Markdown generation.

Required Behavior

1. Prefer local API calls with curl (or any suitable alternative tools).
2. Assume no authentication is required unless the service behavior changes.
3. Start with POST /crawl to get a job ID.
4. Poll GET /crawl/status/{jobId} until crawl completes.
5. Default to Markdown output; use ?format=json only when structured output is needed.
6. Keep crawl scope explicit (limit, depth, subdomain/external toggles, include/exclude patterns).
7. State default crawl behavior: same-domain links only, up to 500 pages per job.
8. If /crawl fails (network, timeout, blocked host), fall back to another method and state the fallback.

Core Commands