vLLM Setup on DGX Station

Deploy a vLLM inference server on DGX Station with validated configuration.

Steps

1. Find the GB300 GPU index. Run:

   nvidia-smi --query-gpu=index,name --format=csv,noheader
   

   Identify the device index for the GB300 (typically device 1). Use this index for --gpus below. Do NOT use --gpus all — mixed coherency will cause CUDA failures.

2. Ask the user which model to serve. If they don't have a preference, suggest:

   • nvidia/Qwen3-235B-A22B-NVFP4 — large MoE model, fits in 279 GB HBM
   • meta-llama/Llama-3.1-70B-Instruct — solid general-purpose model
   • Qwen/Qwen3-8B — small model for testing

3. Check if the user has an HF_TOKEN. Many models require HuggingFace authentication. The token must be passed inline with -e HF_TOKEN="..." — do not rely on shell export in background Docker tasks.