Review Training Data Quality

Test whether the dataset teaches the intended capability rather than merely producing an easy loss curve.

Inspect deterministic distributions

Summarize every product-relevant axis by split and overall: source group, task lane, goal, format, label, abstention, difficulty, candidate count, sequence length, terminal state, role, and preferred-answer position.

Use the generic JSONL summary helper:

python scripts/summarize_jsonl_fields.py corpus.jsonl \
  --field split --field lane --field goal --field label \
  --output quality-distributions.json

Look beyond equal row counts. Verify group-safe splits, distinct source groups, reasonable joint distributions, and enough examples at safety boundaries. Flag any category whose dominance would let the model ignore important context.

Review candidate and label quality