Strategy: Scaling Design

Question: How does performance scale with resources?

Methodology

• Neural Scaling Laws (Kaplan 2020, Hoffmann 2022): Power-law relationships between compute/data/parameters and loss.
• Compute-Optimal Scaling (Chinchilla): Find optimal allocation between model size and data.
• Data Scaling: Characterize learning curves as function of dataset size.
• Model Scaling: Performance vs. parameter count at fixed data.
• Inference Scaling: Throughput/latency vs. batch size, sequence length, model size.

Execution Flow