vLLM Inference Serving

Use this skill to operate vLLM as a production inference server: deploy it with Docker or Kubernetes, configure the model and engine (quantization, tensor parallelism, KV cache, context length), serve the OpenAI-compatible API surface, benchmark throughput and latency with comparable evidence, tune continuous batching, operate the GPUs underneath, and upgrade or roll back safely. This is a tool skill for one named engine. Serving methodology — engine selection, quantization trade-offs, deployment plans, regression triage — belongs to ml-engineering; local single-node GGUF serving with the llama.cpp stack belongs to llama-cpp. This skill owns the day-to-day operation of vLLM itself.

Operating contract

1. Record the deployment before tuning it. Capture the vLLM version or image digest, model and revision, quantization, parallelism, max-model-len, KV cache settings, batching limits, GPU inventory, and workload. The serving config template exists for exactly this.
2. Confirm the target, scope, and rollback path before acting. Read-only discovery (health probes, /metrics, nvidia-smi) may proceed without confirmation. Mutations — restarting a server, changing serving args, scaling replicas, upgrading the image — require an explicit human directive naming the deployment.
3. A server that responds is not a server that serves. /health returning 200 proves liveness, not that the model loaded or that inference works. Verify at the delivery boundary: /v1/models reports the served model and a representative request returns generated tokens.
4. Benchmark before and after every change. vLLM flags, defaults, and behavior change between releases; an unmeasured tuning change is a guess. Compare only matched conditions (version, model, GPU, context, batch, workload) and record the evidence in the benchmark run record.
5. Keep evidence bounded. Summarize logs, configs, and metrics; never dump full server logs, .env files, or HF tokens into chat. --enable-log-requests with debug logging can leak prompt content; keep request logging off or redacted in shared sessions.

The vllm-health script

scripts/vllm-health is an agent-first, read-only probe for a running vLLM server. It issues GET requests only, never mutates, and emits bounded JSON.