Debugging Distributed Hangs in SGLang

Overview

Hangs in distributed inference happen when ranks diverge in state, causing collective operations (AllGather, AllReduce, Broadcast, Barrier) to deadlock. Common causes:

• Size mismatch: ranks pass different tensor sizes to a collective
• Branch divergence: one rank enters a collective, another skips it
• Cascading state drift: a small non-determinism (e.g., floating-point) propagates into different batch structures
• Resource exhaustion: one rank OOMs or crashes, others wait forever

Prerequisites

• py-spy: pip install py-spy or system package. Requires root or CAP_SYS_PTRACE to attach to running processes.
• cuda-gdb: Ships with the CUDA toolkit. Ensure it's on your PATH.

Step 1: Confirm and Locate the Hang

1a. Watchdog / py-spy