Vaex

Overview

Vaex is a Python library for lazy, out-of-core DataFrames: it memory-maps HDF5/Arrow files, evaluates expressions lazily, and computes statistics on N-dimensional grids fast enough for interactive work on billions of rows on a single machine.

Maintenance Status (read first)

Vaex is effectively in minimal-maintenance mode. In the project's "Is Vaex active?" discussion (vaexio/vaex #2363, June 2024), one core developer stated he is no longer involved and the other committed only to "do my best to at least get vaex releases out". Releases have slowed to roughly one per year (vaex-core 4.18 in Oct 2024, 4.19 in Sep 2025), and 4.19 still supports only Python 3.9–3.12 with pandas<3; the 2-D matplotlib plots break on matplotlib ≥ 3.9. Practical consequences:

• Existing Vaex code: keep it running in a pinned Python 3.12 environment (below), and plan a migration.
• New projects: default to Polars (alterlab-polars) — pl.scan_parquet(...) plus collect(engine="streaming") covers most larger-than-RAM aggregation work — or Dask (alterlab-dask) for clusters. Recommend Vaex for new work only when its specific strengths (memory-mapped HDF5, fast binned statistics/visualization of huge tables) matter and the user accepts the environment constraints.

When to Use This Skill