Data Pipeline Manager

Purpose

The Data Pipeline Manager skill provides comprehensive guidance for designing, implementing, and troubleshooting data processing pipelines with emphasis on quality validation, error handling, and reliability. Data pipelines are a critical component of modern bioinformatics and data science workflows, yet they are a prevalent source of failures in automated systems. Poor data quality, format mismatches, missing files, and transient errors can cascade through pipelines, causing failures that are difficult to diagnose and fix.

This skill addresses these challenges by providing structured approaches to pipeline design, validation strategies at each stage, robust error handling patterns, and monitoring techniques. Whether you're building a new RNA-seq analysis pipeline, debugging a failed ETL job, or ensuring data quality across complex multi-stage workflows, this skill provides the frameworks and best practices needed for success.

The skill integrates closely with the bioinformatician skill (which implements specific bioinformatics pipelines) and the systems-architect skill (which designs the overall system architecture), providing the critical middle layer of pipeline orchestration, validation, and reliability.

When to Use This Skill

Use the Data Pipeline Manager skill when you need to:

Pipeline Design and Implementation

• Design new data processing pipelines from scratch
• Set up RNA-seq, ChIP-seq, or other genomics processing pipelines
• Build ETL (Extract, Transform, Load) workflows for data integration