parquet

Apply these conventions when working with Apache Parquet files.

File Structure

"PAR1" magic bytes
[Row Group 1: Column Chunk 1, Column Chunk 2, ..., Column Chunk N]
[Row Group 2: Column Chunk 1, Column Chunk 2, ..., Column Chunk N]
...
File Metadata (Thrift-encoded)
4-byte metadata length (LE)
"PAR1" magic bytes

• Row Group — horizontal partition; each contains one column chunk per column. Processed independently (parallelism unit).
• Column Chunk — all data for one column within one row group. Contiguous in file.
• Page — smallest unit of compression/encoding. Types: Data Page, Dictionary Page, Index Page.
• Metadata is at the end (footer) — enables single-pass writing; readers grab schema/stats from the last few bytes.