Latency Engineering

Comprehensive guidance for diagnosing and reducing latency across the full software/hardware stack.

Mental Model

Latency = time delay between a cause and its observed effect. It is a distribution, not a single number. Always think in percentiles (p50, p95, p99, p99.9). Tail latency dominates real-world user experience far more than averages suggest.

Two fundamental laws:

• Little's Law: Concurrency = Throughput × Latency. Use it to size systems and understand queue dynamics.
• Amdahl's Law: Speedup = 1 / ((1-P) + P/N). Use it to set realistic expectations on parallelization gains.

Decision Framework: Where to Optimize First