Data Formats

Purpose

Design efficient data storage and transfer using Apache Arrow, Parquet, Avro, and ORC formats with appropriate compression, schema evolution strategies, and high-performance transport via Arrow Flight.

Agent Protocol

Trigger

Exact user phrases: "Apache Arrow", "Parquet", "Avro", "ORC", "Arrow Flight", "columnar", "row-oriented", "compression", "schema evolution", "file format", "data format", "columnar storage", "data serialization", "row group", "arrow table", "IPC format", "Flight SQL".

Input Context

Before activating, verify:

• Data access patterns (full scan, column projection, row lookup, point queries)
• Storage target (S3, HDFS, local disk, memory, network transfer)
• Write patterns (append-heavy, overwrite partitions, streaming)
• Processing framework (Spark, DuckDB, pandas, Polars, Dremio, ClickHouse)
• Schema evolution requirements (add/drop/rename columns over time)
• Compression requirements (storage cost vs CPU cost)