Data Import & Validation (Streaming + Error CSV + Metrics + Idempotency)

When to use this skill

Use when working on:

• ETL / import pipelines (CSV/XLSX/JSON) into a DB
• Parsers that currently load whole files into memory
• Data validation/coercion, error isolation, auditability, and reproducibility
• Any import job that must be safe to re-run (idempotent)

Non-negotiables (contract)

1. Process input files sequentially (no “load everything then insert”) to control memory.
2. Validate and coerce types explicitly (define allowed coercions; reject ambiguous cases).
3. Irreparable records must be skipped and logged to an error CSV containing:
   • all original columns exactly as seen in input
   • extra columns: timestamp, file, line, error
4. Emit metrics at minimum: rows_ok, rows_skipped, parse_errors.
5. DB writes must be idempotent: re-running the import must not duplicate or corrupt data.