Distributed Training - Systems Performance Skill

Operational focus: picking and implementing the right parallelism strategy, not the theory. Covers data parallelism through FSDP/ZeRO/tensor+pipeline parallelism, memory-efficient attention, mixed precision at scale, activation checkpointing, rented-GPU cost discipline, and reproducing GPT-2 124M as the canonical sanity check.

Profile before you scale. Debug on the smallest GPU that fits. Stop the instance when done.

ASCII Flow

single GPU (debug/prototype)
  └─ DDP: replicate model, all-reduce gradients — linear scale up to ~8 GPUs
      └─ FSDP2 / ZeRO: shard optimizer state, gradients, params across GPUs
          └─ tensor parallelism: split weight matrices across GPUs (intra-node)
              └─ pipeline parallelism: split layers across nodes (inter-node)
                  └─ context parallelism: shard the sequence dim (long context)
                      └─ expert parallelism: route MoE experts across GPUs (all-to-all)
                          └─ N-D parallelism: DP + TP + PP + CP + EP (frontier MoE)