Data Lineage Mapper

Extracts and maps data lineage from various sources to provide comprehensive data flow visibility.

Overview

This skill parses and extracts data lineage information from SQL queries, dbt projects, Airflow DAGs, and Spark jobs. It generates comprehensive lineage graphs showing data flow from source to destination, enabling impact analysis and data governance.

Capabilities

• SQL parsing for lineage extraction - Parse SELECT, INSERT, MERGE statements
• dbt lineage integration - Extract lineage from manifest.json
• Airflow task lineage mapping - Map data flows across DAG tasks
• Spark job lineage extraction - Parse Spark SQL and DataFrame operations
• Cross-system lineage connection - Connect lineage across different tools
• Column-level lineage tracing - Track individual column transformations
• Impact analysis - Downstream/upstream impact assessment
• Lineage graph generation - Visual and machine-readable lineage
• Integration with data catalogs - Export to DataHub, Amundsen, Alation