Data Distributed Compute

Purpose

Design and tune distributed compute systems for large-scale data processing. Select the right framework (Spark, Dask, Ray, MapReduce), configure YARN/K8s resource management, optimize shuffle and data locality, and tune executors for throughput.

Agent Protocol

Trigger

Exact user phrases: "Hadoop MapReduce", "Spark", "Dask", "Ray", "YARN", "cluster computing", "resource manager", "shuffle", "data locality", "executor", "worker", "task scheduling", "distributed compute", "cluster mode", "dynamic allocation", "speculative execution".

Input Context

Before activating, verify:

• Compute framework preference (Spark, Dask, Ray, MapReduce)
• Data size and shape (TB per run, row counts, join complexity)
• Cluster size and resource per node (cores, memory, network)
• Workload type (batch ETL, ML training, real-time inference, iterative algorithms)
• Storage backend (HDFS, S3, local SSD)
• Scheduling layer (YARN, K8s, standalone)