crawl-site
crawl-site
Crawl a website using Crawlio. Configures settings based on site type, starts the crawl, monitors progress, and reports results.
When to Use
Use this skill when the user wants to download, mirror, or crawl a website for offline access, analysis, or archival.
Workflow
1. Determine Site Type
Before configuring settings, identify the site type. Ask the user or infer from context:
| Site Type | Indicators | Recommended Settings |
|---|---|---|
| Static site | HTML/CSS, no JS frameworks | maxDepth: 5, maxConcurrent: 8 |
| SPA (React, Vue, etc.) | JS-heavy, client-side routing | maxDepth: 3, includeSupportingFiles: true, consider using crawlio-agent for enrichment first |
| CMS (WordPress, etc.) | /wp-content/, admin paths |
maxDepth: 5, excludePatterns: ["/wp-admin/*", "/wp-json/*"] |
More from crawlio-app/crawlio-plugin
audit-site
Use this skill when the user asks to "audit a site", "analyze a website", "review a site", "site health check", or wants a comprehensive analysis including technology stack, issues, and recommendations. Orchestrates a full crawl, enrichment capture, observation analysis, and findings report.
11observe
Use this skill when the user asks to "check observations", "what did Crawlio see", "show crawl timeline", "query the observation log", or wants to review what happened during a crawl session. Queries the append-only observation log with filtering by host, source, operation, and time range.
9finding
Use this skill when the user asks to "create a finding", "record an insight", "what findings exist", "show findings", or wants to create or review evidence-backed analysis insights from crawl observations. Creates and queries curated findings with evidence chains.
8crawlio-mcp
Complete reference for the Crawlio MCP server — 37 tools, 6 code-mode tools, 4 resources, 4 prompts. Use this skill when orchestrating website crawling, export, enrichment, or analysis via Crawlio MCP.
1web-research
Use this skill when the user asks to "research a site", "compare sites", "analyze technology", or wants structured evidence-based web research. Teaches the acquire-normalize-analyze protocol using CrawlioMCP's composite analysis tools.
1extract-and-export
Use this skill when the user asks to "download and export a site", "crawl and extract content", "archive a website", "export as WARC/ZIP/PDF", or wants a complete crawl-extract-export pipeline. Crawls the site, extracts structured content, and exports in the requested format.
1