website-crawler
Website Crawler
High-performance web crawler with TypeScript/Bun frontend and Go backend for discovering and mapping website structure.
When to Use
Use this skill when users ask to:
- Crawl a website or "spider a site"
- Map site structure or "discover all pages"
- Find all URLs on a website
- Generate sitemap or site report
- Analyze link relationships between pages
- Audit website coverage or completeness
- Extract page metadata (titles, status codes)
Keywords: crawl, spider, map, discover pages, site structure, sitemap, all URLs, website audit
Quick Start
More from leobrival/serum-plugins-official
web-crawler
High-performance Rust web crawler with stealth mode, LLM-ready Markdown export, multi-format output, sitemap discovery, and robots.txt support. Optimized for content extraction, site mapping, structure analysis, and LLM/RAG pipelines.
14media-processor
Universal media processor that auto-detects and processes images or videos with appropriate tools. Use when users ask to process media files, batch convert media, compress mixed folders, auto-detect file types, or handle both images and videos together.
3image-processing
Smart image processing for resize, compress, and convert operations with batch support. Use when users ask to resize images, compress photos, convert to WebP/AVIF, batch process images, optimize for social media, or create image thumbnails. Supports modern formats and aspect ratios.
3video-processing
Smart video processing for compress, convert, trim, and audio extraction with batch support. Use when users ask to compress videos, convert to MP4/WebM, extract video clips, trim videos, remove audio, extract audio tracks, or batch process video files. Supports modern codecs and social media presets.
3gif-creation
Create optimized GIFs from videos or image sequences using 2-pass palette generation. Use when users ask to create GIF, video to GIF, optimize GIF, animated GIF, GIF from images, or reduce GIF file size. Supports quality presets and custom timing.
3audit-methodology
Comprehensive audit methodology for web applications covering accessibility (RGAA 4.1), security (OWASP Top 10), performance (Core Web Vitals), and eco-design. Use when users need guidance on audit processes, testing methodologies, compliance standards, or audit best practices. Includes detailed reference documentation for each audit domain.
3