roofline-perf
Installation
SKILL.md
Performance
Every task below runs as task -t ${CLAUDE_SKILL_DIR}/../roofline-toolbox/Taskfile.yml <task> or through an include of that Taskfile in the project one (task infer:<task>). INFER_URL and INFER_PROFILE come from the environment, from arguments (task serve INFER_PROFILE=x) or from inference.env in the current directory.
Metrics and targets
| Metric | What it measures | What sets it | Starting target for coding agents |
|---|---|---|---|
| TTFT | time to first token | prefill (compute-bound) plus queue | p95 under 2 s on a 20k-token prompt with a warm prefix cache |
| TPOT (ITL) | time between tokens | decode (memory-bound): every token reads all the weights | p95 under 50 ms, that is 20 tokens/s or more per stream |
| TPS per user | stream speed | inverse of TPOT | 20 to 40 tokens/s |
| TPS total | throughput | batch size, memory bandwidth | higher with a larger batch, paid for in TPOT |
| p50/p95/p99 | distribution | queue and long prompts | decide on p95, not on the mean |
| e2e | what the client sees | plus network, proxy, queue | large e2e on a fast server points at the proxy |
Formulas and definitions: references/metrics.md. Current numbers: task metrics (histograms since server start); under load: task bench.