Parsing Changelogs

How the Releases fetch pipeline converts changelog pages into structured release data.

Pipeline Overview

The fetch pipeline follows this priority order:

1. Feed adapter — if the source has a known feed URL (in metadata.feedUrl), fetch and parse the feed directly. Fastest and most reliable.
2. Markdown fetch — if metadata.markdownUrl is set, fetch raw markdown instead of rendered HTML.
3. Fast fetch (static providers) — for providers known to serve pre-rendered HTML (Docusaurus, VitePress, WordPress, Ghost, Mintlify), fetch without headless browser rendering. Uses Cloudflare crawl API with render: false. ~10-30x faster than full rendering. Controlled by provider staticContent hint or per-source renderRequired metadata.
4. Cloudflare rendering — for JS-heavy pages (React SPAs, Notion, etc.), use Cloudflare's browser rendering API to get the fully-rendered HTML. Fallback when fast fetch returns no content.
5. Firecrawl monitoring — for sources behind a Cloudflare Managed Challenge that blocks even browser rendering (some vendor help/docs pages, e.g. OpenAI's), an external Firecrawl monitor scrapes the page on a schedule and POSTs changes to the backend, which extracts them through the same parse pipeline. This is a backend-only fetch backend, not a CLI fetch path: it's enabled per source via metadata.firecrawl through the admin API (POST /v1/sources/:slug/firecrawl/sync { enabled: true }), not via metadata edits (that only patches the DB column and skips monitor creation). See docs/architecture/firecrawl-monitoring.md.

After fetching content, the pipeline parses it:

• Incremental parsing — if the source already has releases in the database, extract only new ones by comparing against known releases. This is the default for subsequent fetches.
• Bulk parsing — parse the entire page into releases. Used on first fetch or when --full is specified.