Website Deep Crawl

Input: one or more start URLs (+ optional scope, depth, page-count, globs, removal selectors). Output: per-page records {url, crawl, metadata, text, markdown, html, outboundLinks} for every page reached within scope.

Language

All process output to user (progress updates, process notifications) follows the user's language.

Objective

From a small set of start URLs, breadth-first crawl every reachable in-scope page, strip boilerplate (navigation, header, footer, cookie banners, etc.), and emit per-page LLM-ready content (text / markdown / HTML) plus structured metadata — suitable for feeding RAG pipelines, vector databases, or chatbot knowledge bases.

Prerequisites

• One or more start URLs are provided by the caller.
• Target pages are publicly reachable, OR the running browser is already logged in for any pages behind authentication.
• A working directory is available for writing per-page JSON records and the crawl state file.

Pre-execution Checks