jetson-llm-serve
Jetson LLM Serve
Encodes the Jetson AI Lab GenAI tutorial: on Orin JetPack 7.2 / L4T r39+, use upstream vLLM 0.20+ (vllm/vllm-openai:latest); on older Orin, pick the NVIDIA-AI-IOT prebuilt vLLM container; on Thor, use upstream vLLM 0.20+ or validated native vLLM 0.20+, and use NVIDIA SGLang 26.01 (nvcr.io/nvidia/sglang:26.01-py3, SGLang 0.5.5.post2) when SGLang is requested. Set MAXN, make Hugging Face credentials/cache available, and launch an OpenAI-compatible server. Works for both LLMs and VLMs.
Purpose
Provide a Jetson-appropriate serving recipe for an LLM or VLM using vLLM or SGLang, including runtime path, launch command, endpoint, and verification step.
When to use
- "Run / serve / host this model on a Jetson."
- "Start a vLLM server I can hit from Open WebUI / my app."
- After
jetson-inference-mem-tuneproduced launch flags and the user wants to actually start the server.
For recipe-only questions, answer from this document without starting containers. Run live pre-flight checks only when the user asks you to check this device or execute the deployment.