Zarr DataLoader Performance Tuning

Use this skill when optimizing nvalchemi Zarr reads or writing stores that will later be read through the nvalchemi DataLoader.

Current API model

The pipeline has clean ownership boundaries:

• Reader: storage I/O only. Returns raw CPU tensor dictionaries plus metadata.
• Dataset: validation, optional validation skipping, device transfer, and async prefetch orchestration. Its canonical explicit batch API is load_batches(batch_index_lists).
• DataLoader: sampler/batch iteration, fused prefetch, stream usage, and batch construction.
• MultiDataset: global index composition over multiple Datasets while routing load_batches requests to child datasets.
• Sampler / batch_sampler: semantic sample order and batch membership. Do not rely on sampler windows to optimize storage I/O.