vllm
Installation
SKILL.md
vLLM (local / self-hosted)
Official docs
| Topic | URL |
|---|---|
| vLLM | https://docs.vllm.ai |
| OpenAI server | https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html |
Agent rules
- When Ollama is not enough — multi-user, high QPS, or specific HF weights → vLLM server.
- OpenAI API surface — clients use
OPENAI_BASE_URL=http://host:8000/v1+ dummy key. - GPU memory — check
--gpu-memory-utilization; tensor parallel for multi-GPU. - Harness routing — volume classify / JSON fan-out to local vLLM; implement + review on cloud S-tier.
- Do not commit weights — document model name + HF repo in README, not binaries.