Data Collector

A utility skill to fetch data from remote sources (URLs) and store it locally with rich metadata. It is designed to be the "Extract" part of an ETL pipeline, focusing on reliable data retrieval and change tracking.

Features

• Fetch: Downloads content from HTTP/HTTPS URLs.
• Traceability: Saves a manifest.json tracking source URL, fetch time, and content headers.
• Incremental: Skips download if the remote content hasn't changed (based on Content-Length or ETag/Hash comparison if implemented).

Usage

node data-collector/scripts/collect.cjs --url <URL> --out <output_dir> [options]

Options