coordinated-omission
Coordinated omission
Most load tests carry this bug, and it understates tail latency by orders of magnitude, always in the flattering direction. Named by Gil Tene, creator of HdrHistogram.
A load generator that sends the next request only after the previous response arrives stops sending requests exactly when the system is slowest. The stall is never measured, because no requests were in flight during it. The system is graded only on requests it served promptly.
The load generator coordinates with the system under test to omit the bad results.
The arithmetic
Send one request every 10ms for 100 seconds, so 10,000 requests. The server handles 9,999 in 1ms, then freezes for 100 seconds on the last one.
Closed-loop harness reports: 9,999 samples of 1ms plus one of 100s. p99 = 1ms.
What actually happened: during the freeze, 10,000 requests should have been sent. They would have waited 100s, 99.99s, down to 0. The real median is about 50 seconds.
Same run. One measurement says 1ms at p99, the honest one says 50s at p50.