Mutation testing

The problem with coverage

Line coverage measures which lines ran, not which lines were checked. A test suite calling every function and asserting nothing scores 100%.

def test_calculate():
    calculate(10, 5)      # 100% coverage of calculate(), zero assertions

Branch coverage is better and has the same flaw. Coverage is a necessary condition, not a sufficient one: low coverage proves tests are missing, high coverage proves nothing.

Mutation testing answers the actual question: if the code were wrong, would a test fail?

How it works