DataLad

Overview

DataLad is a data management layer over Git and git-annex. Git tracks the dataset structure, small text files, and the history. git-annex tracks the content of large files, storing each file as a key and keeping the bytes somewhere that is not necessarily the local repository.

That split is the single most important thing to internalise, because it means a freshly cloned dataset contains the full history and the full file listing while containing almost none of the data. A 100 TB dataset clones in seconds and occupies a few megabytes. The bytes arrive only when asked for, per file, with datalad get.

The second thing DataLad adds is provenance. datalad run executes a command and commits the result together with a machine-readable record of the command, its inputs, and its outputs. datalad rerun reads that record back and re-executes it. This turns "how was this figure produced" from an archaeology problem into a command.