Site Data Collector

Explore a site once and get back a ready-to-run Python collector script that pulls all of its data through the local Frevana daemon and Chrome Extension session.

Purpose

This skill runs the Chrome Extension-backed Frevana MCP tool frevana_generate. It:

1. Opens the page in the user's logged-in Chrome and records the site's own XHR/API traffic.
2. Auto-detects the top JSON data endpoint, its pagination parameter, and the fields (deterministic — no LLM).
3. Returns a Python script whose fetch layer is frevana fetch (so every run goes through the user's logged-in browser, at near-zero cost).

It uses a three-layer fallback so most sites produce a working script:

• Layer 1 (JSON API) — the deterministic collector above, when the site has a clean JSON feed that can be re-fetched (a plain GET) through your logged-in browser.
• Layer 1c (Capture mode — signed / SPA APIs) — when the data feed is a signed endpoint (typically POST, with per-request x-s/x-t-style auth or anti-bot headers that the site's own JS computes — e.g. Xiaohongshu's search/notes), a bare frevana fetch replay is rejected (HTTP 406) because only the page can sign its request. frevana generate detects this and emits a capture-mode collector: each run it drives the page with frevana explore (which scrolls and records the site's own XHR in your logged-in browser) and reads the response the page itself signed, then flattens it to CSV (deep-collecting nested image URLs into an image_urls column). Coverage = what one capture window loads (usually the first page, ~20 items); widen it with --duration.
• Layer 2 (DOM) — if there is no JSON API, the tool no longer fails: it returns a DOM collector whose fetch layer is still frevana fetch (raw HTML) and whose selector config defaults to extracting links, with a captured HTML sample embedded in the script.
• Layer 3 (your job) — when you get a Layer-2 script, tighten its ITEM / FIELDS / pagination selector config into the real item rows/columns using that sample (see Execution Order step 6).