ai-scaling-laws

Installation
SKILL.md

AI Scaling Laws — Compute-Optimal Sizing Skill

Use this skill to allocate training compute between model parameters and tokens, then adjust for data and inference constraints.

Quick Reference

Concept Formula / Heuristic Notes
Compute budget C ≈ 6 N D D = training tokens. Say which N you use: Kaplan counts non-embedding params; Chinchilla counts total params (embeddings included). ≈6 = forward + backward matmuls only; the attention term is extra (see compute-budget reference)
Chinchilla-optimal ratio D ≈ 20 × N From Hoffmann et al. 2022; holds compute constant
Kaplan (2020) allocation N ∝ C^0.73, D ∝ C^0.27 No fixed D/N ratio: the compute-efficient path grows D ∝ N^0.37, so D/N falls as C grows. Difference from Chinchilla is methodological (FLOP counting, warmup, optimizer tuning), not simply "wrong" — see post-Chinchilla ref
Optimal N given C N* ≈ (C / 120)^0.5 Derived, not a paper table: it is C = 6ND with D = 20N
Optimal D given C D* ≈ (C / 0.3)^0.5 Same algebra (D* = 20 N*)
Over-training (Llama-style) D ≫ 20 × N Trades higher training loss for cheaper inference. Meta reports over 15T pretraining tokens for Llama 3 8B, far above 20:1. For MoE models apply the ratio to activated, not total, params
GPT-3 (175B) training tokens ~300B tokens ≈1.7 tok/param (Kaplan-era; undercooked by Chinchilla standard)
Chinchilla (70B) training tokens ~1.4T tokens 70B × 20; compute-matched to Gopher (280B, 300B tokens), about 1.9× GPT-3's compute
GPT-2 (124M) at 20:1 ≈2.5B tokens See worked example below; param counts live in ai-pretraining's reference card
Fit constants Hoffmann eq.: α≈0.34, β≈0.28 Besiroglu et al. 2024 refit: α≈0.35, β≈0.37; the published fit implies far more than 20 tok/param, the refit restores ≈20
Installs
18
GitHub Stars
102
First Seen
Aug 12, 2026