Firecrawl Crawl Websites

Mode and routing

• run defaults to asynchronous /start-crawl, then polls. start returns the job immediately. --mode sync calls /crawl, which waits on the server for a bounded period and may still return a pending task; run then continues querying that same task.
• Keep crawl scope explicit: use requested limit, includePaths, excludePaths, and maxDiscoveryDepth. Ask for a scope/budget if the requested collection is unbounded; do not invent a complete-site guarantee.
• sitemap=include|skip|only is a crawl option, not a separate skill. only includes the starting page.
• Async start excludes top-level timeout and pollInterval. For sync crawl these are seconds (timeout default 120, maximum 300). scrapeOptions.timeout is milliseconds. Set client --http-timeout and --wait-timeout above the chosen synchronous server wait.

Examples

bash {baseDir}/scripts/firecrawl.sh run --url 'https://example.com/docs' --json '{"limit":25,"includePaths":["/docs/.*"],"scrapeOptions":{"formats":["markdown"]}}'
bash {baseDir}/scripts/firecrawl.sh start --url 'https://example.com' --json '{"sitemap":"only","limit":25}' --idempotency-key docs-crawl-001
bash {baseDir}/scripts/firecrawl.sh run --mode sync --url 'https://example.com/docs' --json '{"limit":5,"timeout":30}' --http-timeout 40 --wait-timeout 90

Authentication and execution