ray-llm
Ray LLM
Library-reference skill for LLM workloads on open-source Ray — 13 rules across 5 categories covering ray.serve.llm (OpenAI-compatible, vLLM-backed serving) and ray.data.llm (batch inference). This surface churned faster than any other part of Ray — a standalone repo was absorbed and archived, entry points were renamed, and config shapes restructured — so the examples a model learned from mostly no longer run. Each rule names the wrong default it corrects; there is no rule for things a capable model already gets right.
Scope is the LLM-specific layer. Generic Serve/Data/cluster decisions (deployment lifecycle, autoscaling semantics, KubeRay) are the sibling ray skill — the two compose.
Pinned to ray 2.57.0 (ray[llm] extra, which pins its matching vLLM). API claims were verified against the unpacked 2.57.0 wheel and the installed package source, and every config example in the rules was constructed under CPU-only pydantic validation (including the traps, which fail exactly as described); engine/GPU runtime behavior is source-verified only — no model was actually served.
When to Apply
- Standing up or reviewing an OpenAI-compatible LLM serving deployment on Ray
- Writing batch LLM inference over datasets — summarization, embedding, scoring at scale
- Sizing or placing multi-GPU models — tensor/pipeline parallelism, accelerator selection
- Scaling LLM deployments — replica autoscaling, ingress sizing, request routing
- Serving families of LoRA fine-tunes of a shared base model
- Migrating code that uses the archived ray-llm repo, hand-rolled vLLM engines, or pre-2.5x
ray.data.llmnames