Elasticsearch File Ingest

Stream-based ingestion and transformation of large data files (NDJSON, CSV, Parquet, Arrow IPC) into Elasticsearch.

Features & Use Cases

• Stream-based: Handle large files without running out of memory
• High throughput: 50k+ documents/second on commodity hardware
• Formats: NDJSON, CSV, Parquet, Arrow IPC
• Transformations: Apply custom JavaScript transforms during ingestion (enrich, split, filter)
• Batch processing: Ingest multiple files matching a pattern (e.g., logs/*.json)
• Document splitting: Transform one source document into multiple targets

Prerequisites

• Elasticsearch 8.x or 9.x accessible (local or remote)
• Node.js 22+ installed

Setup