Hugging Face Local Models

When to Use

Use this skill when you need use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUFs, quant selection, running servers, exact GGUF file lookup, conversion, and OpenAI-compatible local serving.

Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with llama-cli or llama-server.

Default Workflow

1. Search the Hub with apps=llama.cpp.
2. Open https://huggingface.co/<repo>?local-app=llama.cpp.
3. Prefer the exact HF local-app snippet and quant recommendation when it is visible.
4. Confirm exact .gguf filenames with https://huggingface.co/api/models/<repo>/tree/main?recursive=true.
5. Launch with llama-cli -hf <repo>:<QUANT> or llama-server -hf <repo>:<QUANT>.
6. Fall back to --hf-repo plus --hf-file when the repo uses custom file naming.
7. Convert from Transformers weights only if the repo does not already expose GGUF files.

Quick Start