The Agent Skills Directory

[INDIRECT_PROMPT_INJECTION]: As a web crawler, the skill's core function involves ingesting data from untrusted external websites, presenting a significant attack surface for indirect instructions targeting the agent.
Ingestion points: Content from target URLs is fetched and processed in src/light_crawl/capture.py and src/light_crawl/probe.py.
Boundary markers: The skill includes explicit instructions in SKILL.md (Rule 7) and references/safety.md to treat all page content as untrusted and to never follow instructions found in HTML, JS, or API responses.
Capability inventory: The skill has the ability to execute generated Python code via subprocess and perform network operations.
Sanitization: Employs content cleaning (e.g., Scrapling's --ai-targeted mode) to strip potentially malicious boilerplate or hidden nodes before the agent processes the data.
[DYNAMIC_EXECUTION]: The skill dynamically executes Python code generated by the agent or user to perform acquisition verification and production crawls.
Verifiers: src/light_crawl/validation.py uses subprocess.run to execute scripts located in specific directories (repro/, spider/, or fixtures/).
Site Packs: src/light_crawl/site.py runs verify.py and spider logic from promoted site packs to confirm data integrity.
[PERSISTENCE]: The skill provides optional functionality to maintain long-term presence via system tasks.
Mechanism: scripts/schedule-drift.ps1 allows the registration of daily Windows Task Scheduler tasks (light-crawl-drift) to monitor scraper health.
User Consent: This feature is documented as an opt-in CLI command and is not enabled silently during installation.
[EXTERNAL_DOWNLOADS]: Installation scripts download the skill's source code and third-party dependencies from external repositories.
Repositories: install.ps1 and install.sh download archives and tools from GitHub.
Automation: The installers automate the setup of the Python virtual environment and the installation of Model Context Protocol (MCP) servers.
[COMMAND_EXECUTION]: Frequent use of system shell commands for environment discovery and configuration management.
Environment Doctor: src/light_crawl/doctor.py executes version checks for Node.js, Scrapling, and Git.
MCP Registration: scripts/register_mcp.py executes commands to list and add tools to various AI agent client configurations (Claude Code, Codex, VS Code, etc.).

light-crawl