Firecrawl Known Pitfalls

Overview

Real gotchas when using Firecrawl for web scraping and crawling. Firecrawl handles JavaScript rendering and anti-bot bypassing, but its async crawl model and credit-based pricing create specific failure modes.

Prerequisites

• Firecrawl API key configured
• Understanding of async job patterns
• Awareness of credit-based billing model

Instructions

Step 1: Handle Async Crawl Jobs Properly

crawlUrl returns a job ID, not results. Polling too aggressively wastes credits and may trigger rate limits.

import FirecrawlApp from '@mendable/firecrawl-js';
const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY });