Jetson Inference Memory Tuning

Recommends an inference runtime and the specific memory-related flags to pass to it, given the Jetson SKU/variant and the user's workload. Does not include quantization recipe selection — that lives in the model-benchmarking skill — but it does point at the precision floor each runtime can serve efficiently.

Purpose

Turn a live jetson-memory-audit snapshot into runtime and launch-flag recommendations for LLM/VLM serving on Jetson. Use this when the user needs to fit a model, reduce OOM risk, or switch to a lower-memory serving stack.

When to use

• "Which serving stack should I use on Orin Nano 8 GB to run a 7B model?"
• "vLLM is OOMing — what should --gpu-memory-utilization and --max-model-len be?"
• "Same model, less memory — can I switch from vLLM to llama.cpp?"
• After jetson-memory-audit shows a model server is the top NvMap / PSS consumer.

Prerequisites