Data Pipeline Correctness

Pipelines fail silently: a retry double-counts, a partial write corrupts a table, a schema drift poisons a dashboard, and nobody notices until the numbers are wrong. This skill bakes correctness in at build time. It complements post-hoc data-integrity audit skills (which detect these after the fact) and the Supabase plugin (DB/Edge Functions/RLS).

When this fires

Any job that moves, transforms, or aggregates data: ingestion/ETL/ELT, scheduled aggregations, edge-function workers, pg_cron jobs, queue consumers, webhook processors, backfills, materialized-view refreshes.

Non-negotiables (the 5 that prevent silent corruption)

1. Idempotency — running the same job twice must not change the result. Retries, at-least-once queues, and overlapping cron fires are guaranteed, not hypothetical.

• Use INSERT ... ON CONFLICT (natural_key) DO UPDATE (upsert), not blind INSERT.
• Derive a deterministic dedup key from the source event, not now() or a random id.
• For aggregates: recompute-and-replace a window, or use idempotent deltas — never count = count + 1 on a path that can retry.

2. Atomicity — a job either fully applies or not at all. No half-written batches.

• Wrap multi-row writes in a transaction; stage to a temp/raw table then swap.
• A function that writes to 3 tables must not leave 1 of them updated on failure.