Serving LLMs on AMD Instinct

Get a vLLM endpoint running on AMD Instinct GPU hardware.

Prerequisites

ROCm driver and amd-smi installed on the GPU host
Docker running and accessible (check with docker ps)
/dev/kfd and /dev/dri present on the GPU host
HuggingFace token in HF_TOKEN env var (required for gated models; not required for Qwen3 or Gemma). For gated models (Llama 3.2, Gemma, etc.), the HF token must belong to an account that has accepted the model's license at huggingface.co/<model_id>. A valid token without license acceptance will fail with an opaque "Engine core initialization failed" error.
For remote GPU: SSH key access configured (ssh <user>@<host> must work without a password prompt). If only password access is available, set up keys first: ssh-copy-id <user>@<host>