Dask

Dask provides distributed computing in Python, mirroring familiar NumPy, Pandas, and Scikit-learn APIs with dynamic task scheduling, high-performance shuffles, and GPU acceleration.

When to Use

• Larger-than-Memory Tabular Computation: Processing 10GB-1TB CSV/Parquet datasets on a single machine or multi-node cluster.
• Parallelizing Custom Python Workflows: Using dask.delayed to parallelize arbitrary Python loops and task graphs.
• Distributed Machine Learning: Integrating with Scikit-learn, XGBoost, and LightGBM across distributed worker nodes.
• NumPy & Pandas Scaling: Drop-in familiar APIs for out-of-core array and DataFrame calculations.

Quick Start

import dask.dataframe as dd

# Read multi-gigabyte partitioned CSV files lazily
df = dd.read_csv('data/transactions_*.csv')