Denis' Micro Blog

So...

Recently, I've been refreshing my knowledge of various testing methodologies. Therefore, you get for today an introduction to Mutation Testing, as a way to brain dump.

Okay. Let's get started. 😅

We start with the following example. We have a small class with only one method that takes two integers as input and calculates their quotient and remainder upon division. However, division cannot be performed when the denominator is zero.

To handle this, we added an "if"-condition to check for a zero denominator in our code. Now, we think branch coverage is our most important criterion, and wrote two test cases for this. The first test checks the case where the denominator is not zero, while the second one tests the scenario where the denominator is zero.

Since we achieve 100% branch and statement coverage, we could conclude that these test cases are adequate based on coverage. However, in terms of fault detection capability, we have to admit that our test suite is inadequate because assertions are missing. We can even modify the production code in any way, and the two tests will still pass. For instance, we can replace division with multiplication without altering the first test execution result. Tough luck... 😌

This shows us, that we need to have high code coverage, as well as proper assertions to get better tests.

To check for this principle we can insert artificial defects in the production code to measure the quality of our test code. These artificial defects are usually called mutants. Based on this idea, a robust test suite is one where at least one of its test cases fails when executed against a mutant.

Cool... Let's expand on this idea. How can we change our code to effectively generate those mutants?

Simple. We shouldn't do those kinds of changes which are super weird but rather do those which mimic human error, like small typos or logical errors.

Examples include things like:

• Arithmetic operator errors. Swapping around "+", "-", "*", "/" for example.
• Relational operator errors, like changing up "<=", and ">=".
• Conditional operator errors, where we change things like "&" or "||" around.
• Assignment operator errors, in which we swap between different variable assignment operators, like "=" or "+=".
• Variable replacement errors, where we simply change variables to entirely different ones.

Now... How can we put all of this into practice? asked nobody...

The first step is generating mutants using the mutation operators. Here we have to keep in mind that those mutants have to be adequate and not just any old mutation. Also, different mutants can actually result in equivalent mutations. So keep that in mind. 😉

Then, the test code with each mutant was applied singularly. Nothing new here...

Upon test execution, the mutants are classified into two categories.
A mutant is killed if the test suite fails when executed against the mutant while it passes on the original program. Instead, a mutant is alive when the test suite passes on both the mutant and the original program. An adequate test suite should kill as many mutants as possible. However, killing mutants is not always possible. 😅

Based on the ratio of killed and alive mutants, the quality of the test suite can be measured using the mutation score. This mutation score is defined as the ratio between the number of killed mutants onto the total number of mutants. 

Finally, we analyze the test results with this mutation score and measure the "code quality" this way and improve our code. that took long... 😅

Great. I think we've covered all of the basic things around Mutation Testing. We could go into further detail on what frameworks you could use, or how we could optimize all of this. I think that serves as an intro more than enough.

Hope you have a nice day. ❤

See ya.