inference-performance
Installation
SKILL.md
LLM Inference Performance
Serving an autoregressive transformer is two workloads wearing one trench coat. Prefill processes the whole prompt at once; decode emits one token at a time. They have opposite bottlenecks, so a config tuned for one starves the other. Master this split and the rest (roofline, batching, paging) follows.
The two phases
| Prefill | Decode | |
|---|---|---|
| Work | Process all P prompt tokens in parallel |
Generate token t from tokens 0..t-1 |
| Matmul shape | [P, d] × [d, d] — tall, fat GEMM |
[1, d] × [d, d] — GEMV per request |
| Bottleneck | Compute (FLOPS) | Memory bandwidth (HBM) |
| Passes per token | 1 (amortized over P tokens) |
1 full weight read per token |
| KV cache | Writes P tokens of K,V |
Reads all prior K,V, appends 1 |
| Drives | TTFT (time to first token) | TPOT/ITL (time per output token) |
| Parallelism | Across the sequence (free) | Across the batch (must aggregate) |
| Scales with | Prompt length P (∝ FLOPs, ∝ P² attn) |
Output length O (∝ steps) |
The asymmetry is the whole game. Prefill does ~P tokens of work in one weight-load, so it is compute-bound the moment P exceeds a few dozen. Decode loads all model weights to produce one token, so at batch 1 it is brutally memory-bound — the GPU's ALUs sit idle waiting on HBM. You fix them with different levers: prefill wants more FLOPS and work-splitting; decode wants more bandwidth and a bigger batch.