ES Ingest

[!WARNING] Deprecated: This skill is deprecated in walterra/agent-tools. Use the official Elastic replacement: elasticsearch-file-ingest.

Stream-based ingestion and transformation of large data files into Elasticsearch. Built on node-es-transformer.

Features

• ✅ Stream-based: Handle very large files (20-30 GB tested) without running out of memory
• ✅ High throughput: Up to 20k documents/second on commodity hardware
• ✅ Cross-version: Seamlessly migrate between ES 8.x and 9.x
• ✅ Formats: NDJSON, CSV, Parquet, Arrow IPC
• ✅ Transformations: Apply custom JavaScript transforms during ingestion
• ✅ Reindexing: Copy and transform existing indices
• ✅ Wildcards: Ingest multiple files matching a pattern (e.g., logs/*.json)
• ✅ Document splitting: Transform one source document into multiple targets