ai-data-curation-pretraining

Installation
SKILL.md

Pretraining Data Curation — Functional Reference Skill

Build web-scale and synthetic pretraining corpora, then measure curation choices with controlled ablations.

When to Use This Skill

Activate when the task involves:

  • Finding existing high-quality datasets for pretraining or fine-tuning before building from scratch (see Dataset Discovery reference)
  • Sourcing and filtering CommonCrawl WARCs or other web-scale corpora
  • Implementing or debugging any stage of the curation pipeline above
  • Designing quality filters (heuristic or classifier-based)
  • Running MinHash / LSH deduplication or exact-substring dedup
  • Decontaminating a dataset against evaluation benchmarks
  • Generating synthetic pretraining data (Cosmopedia, Self-Instruct, Evol-Instruct, Nemotron)
  • Designing and executing controlled data ablations
  • Writing datasheets (Gebru et al.) for a curated dataset
  • Understanding open recipe datasets: FineWeb, Dolma, The Pile, RedPajama, SlimPajama, C4, RefinedWeb, OLMo
Installs
19
GitHub Stars
102
First Seen
Aug 12, 2026