Model Serving & Deployment

Reference for taking trained models to production. Follow phases in order.

Phase 0: Compression (before deploying)

Reduce model size and latency before serving. Run benchmarks to confirm tradeoff is acceptable.

uv run ${CLAUDE_SKILL_DIR}/scripts/benchmark_model.py model.joblib model_compressed.joblib data/test.csv

Quantization

• Dynamic (no calibration): torch.quantization.quantize_dynamic — easiest, CPU
• Static (calibration required): torch.quantization.prepare + convert — faster inference
• 4-bit (LLMs): bitsandbytes load_in_4bit=True — 4× memory reduction
• GPTQ: AutoGPTQ — accurate 4-bit for LLMs