Schema Validation Skill

This skill guides the creation of Pandera schemas for validating data in Dagster assets.

Core Principles

1. Pandera + Polars: Use pandera.polars for validation. The project uses Polars for data processing, so validation should align with this.
2. Rich Schemas: All schemas MUST define:
   • Data Types: Explicit Polars data types (e.g., pl.Utf8, pl.Int64).
   • Nullability: Explicit verify nullable=True or nullable=False.
   • Descriptions: A clear description for every field explaining the data's meaning or source.
3. Unified Validation: Use the same schema for validating Bronze (JSON) and Silver (Parquet/Delta) layers where possible.

Validation Strategies

1. Bronze Layer (JSON/Dicts) -> Ad-hoc Validation

For assets that return list[dict] (Bronze layer), do not use pandera_schema_to_dagster_type as it forces a Dagster Type check that expects a DataFrame, adding unnecessary complexity to the asset signature.