Vaex

Vaex is a hidden gem. It uses Memory Mapping to open 100GB files instantly on a laptop and visualize them.

When to Use

• Billion-Row Tabular Analytics on a Single Laptop: Interactive exploration and statistics on 100M-1B+ rows without cloud clusters.
• Zero-Copy Memory-Mapped Files (HDF5 / Arrow): Opening 100GB+ files in milliseconds without loading data into RAM.
• Virtual Columns & Lazy Evaluation: Computing derived features and statistical transforms without allocating additional memory.
• Ultra-Fast 2D Heatmaps & Visualizations: Generating histograms, binned heatmaps, and scatter visualizations instantly.

Quick Start

import vaex

# Instantly memory-map 100M+ row HDF5/Arrow dataset without loading into RAM
df = vaex.open('billion_records.hdf5')