PySpark Test Generator Skill

Overview

This skill enables AI agents to automatically generate comprehensive PySpark-based data quality validation tests for Databricks tables. It creates executable test suites that validate data completeness, accuracy, consistency, and conformity.

Purpose

• Generate PySpark validation tests based on data profiling results
• Create reusable test frameworks for data quality monitoring
• Implement custom validation rules using PySpark SQL and DataFrame operations
• Produce detailed test reports with pass/fail metrics
• Support continuous data quality monitoring in production pipelines

When to Use This Skill