Distributed Training Patterns

Expert patterns for DDP, FSDP, DeepSpeed, and mixed precision training across multiple GPUs.

Pattern 1: PyTorch DDP Training Loop

Standard DDP setup with torchrun launcher, DistributedSampler, and proper cleanup.

# train_ddp.py
import os
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader, DistributedSampler