vLLM serve

One vLLM installation (a uv tool) serves every model a machine runs, each through a profile and a server of its own: started when someone asks for it, shared by everyone who asks while it runs, and stopped after a spell without requests, giving the GPUs back. Installing or upgrading vLLM, and a server that will not start → references/install.md. A machine other people use, or one reached over ssh → references/shared-machines.md, before the first install or up there.

Calling. Start every shell call with this statement, filled in with this file's absolute directory, then the commands: set -o pipefail; export PATH="<this directory>/scripts:$PATH"; — serve.py <word> … is then found from anywhere in the call, your working directory stays where it is, and a failure anywhere in a pipe sets the call's exit code. serve.py -h lists the words and the profile keys; serve.py <word> -h is each word's reference: its lines' fields, its exit codes. stdout is data, one JSON line per profile; a line reporting a problem carries a hint, the next step. Keep stderr: a start's progress, the warning that the last start failed, and on exit 2 a last line {code, detail, hint} saying why nothing was tried.

Use a server

serve.py up <profile>
CUDA_VISIBLE_DEVICES=4,5 serve.py up <profile> --data-parallel-size 2    # GPUs and vLLM arguments for this start

Speak the OpenAI API at url with model as the model name. up waits until the server answers or the start fails: the first up of a profile downloads its weights, and a start takes from one or two minutes (a small model) to several (a 27B model, longest on the first start after an install), so call it in the background for anything interactive. Call up again before each new batch of work: a server you used an hour ago has stopped, and up resets its idle clock. Arguments and environment passed to up shape this start only; a choice that must hold at the next start, which another caller may trigger after an idle stop, belongs in a profile.

When up says more

What to do beyond each line's hint: