using-training-optimization
Using Training Optimization
Overview
This meta-skill routes you to the right training optimization specialist based on symptoms. Training issues often have multiple potential causes—this skill helps diagnose symptoms and route to the appropriate specialist. Load this skill when you encounter training problems but aren't sure which specific technique to apply.
Core Principle: Diagnose before routing. Training issues often have multiple causes. Ask clarifying questions to understand symptoms before routing to specific skills. Wrong diagnosis wastes time—systematic routing saves it.
Knowledge cutoff (2026-05): The optimizer / schedule / precision landscape covered in this pack is calibrated to early-2026. Modern entries (Lion, Sophia, Muon, AdEMAMix, Schedule-Free, Prodigy, AdamW8bit / paged optimizers, FP8, WSD schedules, muP/mu-Transfer, ZeRO/FSDP strategy choice) are all routable. AdamW + cosine schedule + BF16 mixed precision remains the boring-and-correct default for most workloads; modern alternatives are pointed-tool replacements with documented trade-offs, not blanket upgrades. The router does not hardcode model IDs or vendor-specific recipes — capability tiers and symptom signatures drive routing.
Cross-pack frame (frozen vocabulary):
- training-optimization (this pack) covers training dynamics: optimizers, schedules, precision, batch size, gradients, loss objectives, regularization, in-flight experiment tracking.
- yzmir-llm-specialist (
llm-finetuning-strategies.md) owns preference-tuning method choice: DPO, GRPO, SimPO, ORPO, KTO, IPO, RLHF/RLAIF. The boundary: "should I use DPO or SFT for my task?" is a method question → llm-specialist. "Does Lion beat AdamW for my DPO run?" is a training-dynamics question → here (optimizer choice) plus llm-specialist (preference method). - yzmir-pytorch-engineering owns the implementation surface: FSDP API,
torch.compile,torch.amp, DataLoader internals, distributed bring-up. We make the strategy call (e.g. "you need ZeRO-3"); they handle the API. - yzmir-ml-production owns the production registry / lineage. Training-side experiment tracking (in-flight logging, run comparison) is here; long-lived model registry, lineage, and deployment artifacts are there.