ai-scaling-laws
Installation
SKILL.md
AI Scaling Laws — Compute-Optimal Sizing Skill
Use this skill to allocate training compute between model parameters and tokens, then adjust for data and inference constraints.
Quick Reference
| Concept | Formula / Heuristic | Notes |
|---|---|---|
| Compute budget | C ≈ 6 N D | D = training tokens. Say which N you use: Kaplan counts non-embedding params; Chinchilla counts total params (embeddings included). ≈6 = forward + backward matmuls only; the attention term is extra (see compute-budget reference) |
| Chinchilla-optimal ratio | D ≈ 20 × N | From Hoffmann et al. 2022; holds compute constant |
| Kaplan (2020) allocation | N ∝ C^0.73, D ∝ C^0.27 | No fixed D/N ratio: the compute-efficient path grows D ∝ N^0.37, so D/N falls as C grows. Difference from Chinchilla is methodological (FLOP counting, warmup, optimizer tuning), not simply "wrong" — see post-Chinchilla ref |
| Optimal N given C | N* ≈ (C / 120)^0.5 | Derived, not a paper table: it is C = 6ND with D = 20N |
| Optimal D given C | D* ≈ (C / 0.3)^0.5 | Same algebra (D* = 20 N*) |
| Over-training (Llama-style) | D ≫ 20 × N | Trades higher training loss for cheaper inference. Meta reports over 15T pretraining tokens for Llama 3 8B, far above 20:1. For MoE models apply the ratio to activated, not total, params |
| GPT-3 (175B) training tokens | ~300B tokens | ≈1.7 tok/param (Kaplan-era; undercooked by Chinchilla standard) |
| Chinchilla (70B) training tokens | ~1.4T tokens | 70B × 20; compute-matched to Gopher (280B, 300B tokens), about 1.9× GPT-3's compute |
| GPT-2 (124M) at 20:1 | ≈2.5B tokens | See worked example below; param counts live in ai-pretraining's reference card |
| Fit constants | Hoffmann eq.: α≈0.34, β≈0.28 | Besiroglu et al. 2024 refit: α≈0.35, β≈0.37; the published fit implies far more than 20 tok/param, the refit restores ≈20 |