ML Data Pipeline

Overview

Use this skill for data ingestion, validation, preprocessing, feature engineering, dataset versioning, feature stores, batch and streaming pipelines, and data-quality monitoring. In ML, data pipeline correctness is often more important than model sophistication. A pipeline must produce leakage-safe training data and consistent serving features.

Data Data Pipeline Invariants

• Raw data is immutable or snapshot-addressable.
• Schemas, statistics, and quality expectations are validated before training and serving.
• Transformations are versioned and reproducible.
• Splits are created before leakage-prone operations such as oversampling, target encoding, feature selection, or normalization.
• Time-dependent features use only information available at prediction time.
• Offline training features match online serving features.
• Sensitive data is minimized, access-controlled, encrypted, and audited.

Ingestion and Storage

Choose storage based on data shape and access pattern. Object storage with Parquet/Arrow is a strong default for tabular batch ML. Delta Lake, Apache Iceberg, or Hudi add ACID tables, schema evolution, and time travel. Use warehouses for governed SQL features, vector stores for embedding retrieval, and streaming logs for online behavior. Store raw, cleaned, feature, and model-ready layers separately. For Azure Storage pointer blobs used by ADF to pass Azure ML code asset versions, load ml-azureml-adf-automation.