Build a dataset from evidence

Read the operating agreement, data sourcing, and artifact records. For hosted storage use managed guidance.

For bundled local tools and exact commands, read toolkit setup. Run the workspace installer for this skill; it preserves existing customized files. For the managed route, use concrete service operations after confirming the connected schema.

Reuse and curate

Inspect existing cases and ask only what cannot be derived: whether they still match traffic, who verified references, and what decisions the eval must support. Preserve useful cases. Distinguish inputs from reference answers, rubrics, and baseline outputs; expected does not always mean a single correct sentence.

For every case retain source, permitted retention, failure/category tags, conversation or source group, necessary attachments/context, and reference origin. Capture the environment/fixture necessary to replay an agent. Keep source records unchanged.

Use confirmed failures as regression examples, but also include normal and negative cases. A challenge suite and a representative benchmark answer different questions; label them separately. Generate synthetic cases only to fill specific gaps and have a knowledgeable reviewer verify realism.

Protect comparisons

Deduplicate and group related conversations, documents, customers where appropriate, and paraphrase families before splitting. Split by group so related examples do not cross boundaries. Stratify by relevant task/category where enough groups exist; do not select development cases just because their baseline scores are low.