SGLang Setup on DGX Station

Deploy an SGLang inference server on DGX Station with validated configuration.

Steps

1. Find the GB300 GPU index. Run:

   nvidia-smi --query-gpu=index,name --format=csv,noheader
   

   Identify the device index for the GB300 (typically device 1). Use this index for --gpus below. Do NOT use --gpus all — mixed coherency will cause CUDA failures.

2. Ask the user which model to serve. If they don't have a preference, suggest:

   • Qwen/Qwen3-8B — small, fast, good for testing
   • Qwen/Qwen3-32B — medium, good balance
   • meta-llama/Llama-3.1-70B-Instruct — large general-purpose

3. Check if the user has an HF_TOKEN. Pass inline with -e HF_TOKEN="...".