inference-caching-and-kv
Installation
SKILL.md
Inference Caching & KV Cache Management
Three distinct caches get confused under one word. They live at different layers, have different correctness properties, and fail differently.
| Cache | Layer | Match | Returns | Risk if wrong |
|---|---|---|---|---|
| Prompt cache | Provider / serving engine | Exact token-prefix | Skips prefill compute, regenerates output | None (cheaper, identical output) |
| Semantic cache | Your app, in front of the API | Embedding similarity | A stored prior response | Serves a wrong answer |
| KV cache | GPU, inside the model | Per-token K/V tensors | Enables O(1) decode | OOM / preemption |
Rule of thumb: prompt cache is exact + cheap + safe — on always. Semantic cache is fuzzy + risky — use surgically. KV cache is not optional (it's how decode works) and at scale dominates GPU memory.