Yao ChatGPT Crawler

Inputs

Keywords/questions, repeat count, target entity, entity type (person/company/product), browser profile, single-query interval preset, and optional output directory. Competitors must match the target type.

Default interval is a random 30s-1m between fresh samples. For lower request frequency, use 1-3m or 3-10m; --safe-random-delay maps to 3-10m. Slower intervals reduce account-risk pressure but do not guarantee that ChatGPT will not throttle or challenge the session.

Workflow

1. Read references/user-setup-and-usage.md, references/chatgpt-crawl-workflow.md, and references/report-contract.md as needed.
2. Run node scripts/preflight.mjs --profile <profile> before fresh crawling.
3. Stage 1: run scripts/chatgpt_batch_crawl.mjs with questions, repeat, profile, target entity/type, interval preset, and output dir.
4. Stage 2: run scripts/analyze_chatgpt_results.py with target entity/type, optional brands file, optional semantic review, and report output dir.
5. Return crawl JSON, summary, structured Markdown/Excel, HTML report, optional semantic-review cache, and failed logs.

Honest Boundaries