YC Jobs Scraper

This skill provides a robust architecture for scraping jobs from YCombinator and workatastartup.com. It is designed to run automatically, bypass login bottlenecks, and maintain state to never scrape duplicate jobs.

Architecture

The scraper uses a hybrid approach to maximize reliability and minimize bot detection:

1. Authentication: scripts/auth.js uses Playwright to let a human log in once and saves the session to scripts/state.json.
2. Database: scripts/db.js uses better-sqlite3 to manage scripts/jobs.db. It tracks every company_slug and job_id ever seen.
3. Primary Extraction: scripts/scraper.js loads state.json, visits YC query URLs, and extracts company slugs from the hidden Inertia.js data-page JSON payload.
4. Job Extraction (JSON): It then visits the authenticated company pages (/companies/[slug]) to extract jobs from the backend JSON payload to ensure we get the real job_id for accurate deduplication.
5. Job Extraction (Fallback): If the JSON extraction fails, it falls back to parsing public HTML job cards from ycombinator.com/companies/[slug]/jobs.

Workflows