study-scraping
Installation
SKILL.md
Study Scraping
Purpose
Collect public evidence as normalized research.v1 cards. This skill owns extraction, pagination, retries, budgets, and metadata. Hand packets to a downstream topic-research or analysis step for confidence scoring and routing.
Keep raw titles, captions, quotes, handles, dates, IDs, metrics, and URLs in their source language. Never store tokens, cookies, private data, or full licensed content in a packet.
Tool Router
- Use normal web search or connectors for a few known pages.
- Use TikHub for public Instagram, YouTube, or Reddit profiles, posts, videos, transcripts, comments, creator mechanics, and audience language.
- Use Firecrawl to discover web sources, then scrape only selected URLs that can change the research decision.
- Use Scrapling for allowed static/JavaScript pages when local targeted extraction or a repeatable spider is needed.
- Stop on robots/terms restrictions, login walls, private content, paywalls, HTTP 401/403/429, or anti-bot challenges. Do not bypass them.
Read tikhub-social-api.md for routes and normalized fields. Read firecrawl-web-agent.md for broad web collection. Read scrapling-local-extraction.md for allowed local extraction. For Instagram boundaries, read instagram-safe-scrapling-extension.md.