crawl

Installation
SKILL.md

Web Crawling Skill

Overview

This skill enables deep web crawling using hakrawler and gospider to discover subdomains, extract endpoints, analyze JavaScript files, and map web application attack surfaces. The skill provides a unified Python wrapper that combines both tools for comprehensive URL discovery and categorization. All bash command outputs (stdout/stderr) are automatically logged for debugging and analysis.

When to Use This Skill

Use this skill when users request:

  • Crawling or spidering a URL or domain
  • Subdomain discovery and enumeration
  • Endpoint and API discovery
  • JavaScript file extraction and analysis
  • Web application reconnaissance or mapping
  • Bug bounty scoping and enumeration
  • Attack surface discovery

Trigger keywords: crawl, spider, enumerate, discover subdomains, find endpoints, map application, recon, reconnaissance

Installs
2
GitHub Stars
15
First Seen
12 days ago