Bright Data Async Snapshot Pipeline

Overview

Implement the documented trigger, progress, and download lifecycle as a state machine. Keep dataset identifiers and approved inputs separate from the API key; cap polling, validate terminal states, and stream results rather than loading arbitrary payloads into memory.

Prerequisites

• A reviewed dataset ID and approved public input manifest
• A named-user Bright Data API key in the runtime secret manager
• A retention, schema, maximum-record, and maximum-byte policy

Instructions

Step 1: Validate the manifest

Read the requested dataset/inputs and Grep for disallowed target classes or fields. Hash the approved input manifest before submission.

Step 2: Trigger once