Dribl Crawling

Overview

Extract clubs and fixtures data from https://fv.dribl.com/fixtures/ (Cloudflare-protected SPA) using a real browser for Cloudflare clearance, then making direct API calls via page.evaluate(fetch(...)) rather than driving the SPA UI.

Purpose: Crawl dribl.com to maintain up-to-date clubs and fixtures.

Why direct API — not SPA scraping

The SPA's default date-window views (Results tab + Fixtures upcoming) create a dead zone of 4+ rounds whenever a team is regraded to a new league mid-season. The dribl API accepts a round param; iterating round=1..N is deterministic and never drops rounds.

Raw curl returns HTTP 403 (Cloudflare). The browser context has clearance cookies — page.evaluate(fetch(...)) uses them transparently.

Architecture

Data flow: