FireCrawl Reference Architecture

Overview

Production architecture for web scraping and data extraction with FireCrawl. Covers crawl job orchestration, content extraction pipelines, structured data output, and site mapping workflows.

Prerequisites

• FireCrawl API key
• @mendable/firecrawl-js SDK installed
• Queue system for large crawl jobs (optional)
• Storage for extracted content

Architecture Diagram

┌─────────────────────────────────────────────────────┐
│                 Crawl Orchestrator                    │
│  ┌──────────┐  ┌──────────┐  ┌──────────────────┐   │
│  │ Scrape   │  │ Crawl    │  │ Map              │   │
│  │ (single) │  │ (multi)  │  │ (discovery)      │   │
│  └────┬─────┘  └────┬─────┘  └────────┬─────────┘   │