Vaex

Overview

Vaex is a high-performance Python library designed for lazy, out-of-core DataFrames to process and visualize tabular datasets that are too large to fit into RAM. Vaex can process over a billion rows per second, enabling interactive data exploration and analysis on datasets with billions of rows.

When to Use This Skill

Use Vaex when:

• Processing tabular datasets larger than available RAM (gigabytes to terabytes)
• Performing fast statistical aggregations on massive datasets
• Creating visualizations and heatmaps of large datasets
• Building machine learning pipelines on big data
• Converting between data formats (CSV, HDF5, Arrow, Parquet)
• Needing lazy evaluation and virtual columns to avoid memory overhead
• Working with astronomical data, financial time series, or other large-scale scientific datasets

Core Capabilities