scrape

This file is 310 lines long; read all of them.

SKILL_DIR below stands for the absolute path of the directory that contains this file. SKILLS_DIR stands for the directory that holds every skill directory, one of them being SKILL_DIR.

You are orchestrating the full web scraping workflow, from a user's prompt to a working Scrapy spider with web-poet page objects.

Prerequisites

Requires uv. Install if missing.

Input

This is the user prompt: $ARGUMENTS. You need to extract the following information from it:

• url: target website URL. May be a homepage or a specific detail page.
• what: what to extract (e.g. "product", "job listing", "recipe")
• other useful details: user instructions that should affect the plan (e.g. which fields to extract, whether to use browser rendering etc.)