DocETL Pipeline Development

DocETL is a system for creating LLM-powered data processing pipelines. This skill helps you build end-to-end pipelines: from data preparation to execution and optimization.

Workflow Overview: Iterative Data Analysis

Work like a data analyst: write → run → inspect → iterate. Never write all scripts at once and run them all at once. Each phase should be completed and validated before moving to the next.

Phase 1: Data Collection

1. Write data collection script
2. Run it immediately (with user permission)
3. Inspect the dataset - show the user:
   • Total document count
   • Keys/fields in each document
   • Sample documents (first 3-5)
   • Length distribution (avg chars, min/max)
   • Any other relevant statistics
4. Iterate if needed (e.g., collect more data, fix parsing issues)