Benchmarking Pyrefly

Use this skill when asked to benchmark pyrefly performance, run a pyrefly bench, compare cold-start vs error-propagation latency, or update the PyTorch benchmark pin. All benches live in pyrefly/pyrefly/benches/ and use Criterion.

Build mode matters: always build optimized, or the numbers are meaningless. Buck uses @fbcode//mode/opt (or @fbcode//mode/opt-clang-thinlto for final numbers). Cargo cargo bench already uses the optimized bench profile; cargo run/cargo build would need --release. Debug builds run 3-10x slower and are not comparable.

Never run two benchmarks in parallel — they compete for CPU and memory and the timings become unreliable.

Micro benchmarks (fast)

Deterministic, single-threaded microbenchmarks of type-checker internals. Cheap to run, good for a quick signal.