DuckDB - The SQL Engine for Scientific Data

DuckDB brings the power of professional SQL to the Python data science stack. It is optimized for "Online Analytical Processing" (OLAP), meaning it excels at large-scale aggregations, joins, and complex queries on datasets that are larger than memory.

When to Use

• Performing complex SQL queries (JOINs, Window functions) on Pandas or Polars data.
• Querying large Parquet or CSV files directly without loading them into memory.
• Efficiently joining data from different sources (e.g., a CSV file and a Pandas DataFrame).
• Building analytical pipelines where SQL is more concise or faster than DataFrame code.
• Managing local datasets that are too big for Excel but don't need a full PostgreSQL server.
• Intermediate data storage and feature engineering for Machine Learning.

Reference Documentation

Official docs: https://duckdb.org/docs/
Python API: https://duckdb.org/docs/api/python/overview
Search patterns: duckdb.sql, duckdb.query, duckdb.read_parquet, duckdb.from_df