Menu

1 / 5

A Mutation-Testing Study of 50 Python Libraries Found 4 Real Problems

#towardsdatascience#science#code#tests#test
Reading 0:00
15s threshold

Imagine your codebase as a mothership under constant fire in an AI development battle, where every edit, yours or your AI assistant's, can let a bug through. Your test suite is its shield, and every green test run says the shield is up. But green only means nothing has got through yet. The only way to know the shield holds is to attack it on purpose: plant a wide variety of bugs, one at a time, and count how many your tests stop. I planted 2,312 of them in 50 Python libraries (listed at the end). In the young ones, 4 in 10 got through; in the mature ones, fewer than 2 in 10.

Your codebase as a mothership under attack. Image by the author.

My own Python project has over 2,200 tests, all written by an AI assistant. But passing tests can give a false sense of security, since they only tell me that the tests agree with the code as it is today. They do not tell me whether the tests would notice if tomorrow's change broke it.

There is a well-established way to find out, called mutation testing .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.