ai-data-curation-pretraining
Installation
SKILL.md
Pretraining Data Curation — Functional Reference Skill
Build web-scale and synthetic pretraining corpora, then measure curation choices with controlled ablations.
When to Use This Skill
Activate when the task involves:
- Finding existing high-quality datasets for pretraining or fine-tuning before building from scratch (see Dataset Discovery reference)
- Sourcing and filtering CommonCrawl WARCs or other web-scale corpora
- Implementing or debugging any stage of the curation pipeline above
- Designing quality filters (heuristic or classifier-based)
- Running MinHash / LSH deduplication or exact-substring dedup
- Decontaminating a dataset against evaluation benchmarks
- Generating synthetic pretraining data (Cosmopedia, Self-Instruct, Evol-Instruct, Nemotron)
- Designing and executing controlled data ablations
- Writing datasheets (Gebru et al.) for a curated dataset
- Understanding open recipe datasets: FineWeb, Dolma, The Pile, RedPajama, SlimPajama, C4, RefinedWeb, OLMo