crawl
Installation
SKILL.md
Web Crawling Skill
Overview
This skill enables deep web crawling using hakrawler and gospider to discover subdomains, extract endpoints, analyze JavaScript files, and map web application attack surfaces. The skill provides a unified Python wrapper that combines both tools for comprehensive URL discovery and categorization. All bash command outputs (stdout/stderr) are automatically logged for debugging and analysis.
When to Use This Skill
Use this skill when users request:
- Crawling or spidering a URL or domain
- Subdomain discovery and enumeration
- Endpoint and API discovery
- JavaScript file extraction and analysis
- Web application reconnaissance or mapping
- Bug bounty scoping and enumeration
- Attack surface discovery
Trigger keywords: crawl, spider, enumerate, discover subdomains, find endpoints, map application, recon, reconnaissance