Data Batch Processing

Purpose

Design efficient batch processing architectures using Hive, Spark SQL, and optimized file formats. Master partitioning, bucketing, Catalyst optimizer tuning, vectorized reads, file format selection, and dynamic partition pruning for large-scale analytical workloads.

Agent Protocol

Trigger

Exact user phrases: "batch processing", "Hive", "Spark SQL", "Pig", "HQL", "Hive partition", "Spark partition", "bucketing", "query optimization", "ORC", "Parquet", "Avro", "vectorized read", "dynamic partition pruning", "Catalyst optimizer", "Tungsten", "Hive metastore", "reduce tasks".

Input Context

Before activating, verify:

• Query engine (Hive on Tez, Hive on MR, Spark SQL, Presto, Trino)
• File format currently used (text, Parquet, ORC, Avro, JSON)
• Table volume (row count, size in TB, partition count)
• Partition column(s) and cardinality
• Common query patterns (full scan, filtered, aggregated, joined)
• Cluster resources (cores, memory, number of nodes)