Quantization & Model Compression

Compression is how a model that trained on a cluster serves on the GPU you can afford. Three orthogonal tools attack three different costs: quantization cuts memory/cost (maybe lossy), speculative decoding cuts decode latency (lossless), distillation produces a genuinely smaller model (training cost up front). Pick by the bottleneck, not by hype. They compose.

1. Why compress

• Memory fit. A 70B model in FP16 weights = ~140 GB — won't fit one 80 GB H100. At INT4 it's ~35 GB and fits with room for KV cache. Quantization is often the only way onto one GPU.
• Cost. Fewer/smaller GPUs per replica; higher batch density. Half the bytes per weight ≈ half the VRAM ≈ roughly half the $/token at fixed throughput.
• Latency. LLM decode is memory-bandwidth bound, not compute bound — every token streams all weights from HBM. Halving weight bytes ~halves the per-token read, so INT4/INT8 decode is faster even when the math is the same. Speculative decoding attacks the same bottleneck by verifying many tokens per forward pass.
• Throughput / batch. Smaller weights + smaller KV cache leave headroom for larger batches, which is where serving economics live.

Rule: prefill is compute-bound (FLOPs), decode is bandwidth-bound (byte reads). Quantization helps decode most; it does little for prefill-heavy/long-prompt workloads.

2. Numeric formats