Setup Local LLM

Get a local model ready for text-corpus work.

Procedure

1. Audit: check for ollama, llama.cpp, vllm, LM Studio, llamafile. Check GPU (nvidia-smi, rocm-smi) for VRAM budget.

2. Recommend by task profile:

   Task profile	Suggested models
   Classification / labeling into known categories (1-3k tokens, short answer)	llama3.1:8b, qwen2.5:7b, gemma2:9b — 8-16GB VRAM
   Semantic judgment on longer docs	qwen2.5:14b, llama3.1:70b (quantized) — 24GB+ VRAM
   Embedding generation	nomic-embed-text, mxbai-embed-large via Ollama, or all-MiniLM-L6-v2 via sentence-transformers (CPU-fast)
   Topic labeling (one-off, 20-50 calls)	Any 7B+ instruct model