Dask

Dask scales Python. It looks like Pandas/NumPy but runs on clusters. 2025 updates focus on High Performance Shuffle and GPU integration.

When to Use

• Big Data: When data > RAM but < BigQuery scale.
• Cluster Computing: Utilizing a Kubernetes cluster for Python functions.
• Xarray: Backend for geospatial data.

Core Concepts

Collections

dask.dataframe, dask.array, dask.bag.

Scheduler

Decides where to run tasks (Local Threads, Processes, or Distributed Cluster).