Mutation testing concept (PIT) in JUnit - Build an Automation Script

Verify mutation testing detects a simple mutation in Calculator add method

Preconditions (3)

Step 1: Run the CalculatorTest normally to verify it passes

Step 2: Modify the Calculator add method by changing '+' to '-' operator (simulate mutation)

Step 3: Run the CalculatorTest again

Step 4: Observe if the test fails due to mutation

✅ Expected Result: The test should fail after mutation, showing PIT detected the mutation

Automation Requirements - JUnit 5

Assertions Needed:

Assert that Calculator add method returns correct sum before mutation

Assert that test fails after mutation is introduced

Best Practices:

Use clear and simple test methods

Keep production and test code separate

Use PIT plugin or command line to run mutation tests

Do not hardcode mutation, rely on PIT tool to generate mutations

Automated Solution

JUnit

import static org.junit.jupiter.api.Assertions.assertEquals;
import org.junit.jupiter.api.Test;

class Calculator {
    public int add(int a, int b) {
        return a + b; // Original code
    }
}

public class CalculatorTest {

    @Test
    void testAdd() {
        Calculator calc = new Calculator();
        assertEquals(5, calc.add(2, 3), "2 + 3 should equal 5");
    }
}

/*
To perform mutation testing with PIT:
1. Run the CalculatorTest normally - it should pass.
2. Configure PIT in your build tool (Maven/Gradle) or IDE.
3. Run PIT mutation testing - it will automatically mutate the '+' operator to '-'.
4. PIT will run tests against mutated code.
5. If testAdd fails on mutated code, mutation is detected (killed).
*/

The Calculator class has a simple add method that sums two numbers.

The CalculatorTest class tests that add(2,3) returns 5.

Mutation testing tools like PIT automatically change code (e.g., '+' to '-') to check if tests catch errors.

If the test fails after mutation, it means the test is good and detects faults.

This example shows how to write a test that PIT can use to verify mutation detection.

Common Mistakes - 3 Pitfalls

Hardcoding mutations manually in production code

{'mistake': 'Not running mutation tests separately from normal tests', 'why_bad': "Mutation tests require special tool runs; running normal tests only won't detect mutations.", 'correct_approach': 'Use PIT plugin or command line to run mutation tests explicitly.'}

Writing tests that do not assert meaningful outcomes

Bonus Challenge

Now add tests for a subtract(int a, int b) method and verify mutation testing detects mutations in it

Show Hint

Practice

1. What is the main purpose of mutation testing using PIT in JUnit projects?

easy

A. To check if tests can detect small changes (mutations) in the code

B. To improve the performance of the application

C. To generate new test cases automatically

D. To refactor the code for better readability

Solution

1. Step 1: Understand mutation testing concept

   Mutation testing changes code slightly to see if tests catch the changes.

2. Step 2: Identify PIT's role in JUnit projects

   PIT automates mutation testing to check test effectiveness.

3. Final Answer:

   To check if tests can detect small changes (mutations) in the code -> Option A

4. Quick Check:

   Mutation testing = detect code changes [OK]

Hint: Mutation testing checks test strength by code changes [OK]

Common Mistakes:

• Thinking mutation testing improves app speed
• Confusing mutation testing with test generation
• Assuming mutation testing refactors code

2. Which of the following is the correct way to run PIT mutation testing with Maven in a JUnit project?

easy

A. mvn test:pitest

B. mvn pitest:mutationCoverage

C. mvn mutation:test

D. mvn pit:run

Solution

1. Step 1: Recall PIT Maven command

   The standard Maven command to run PIT is mvn pitest:mutationCoverage.

2. Step 2: Check other options for correctness

   Other options are invalid or do not match PIT's official command syntax.

3. Final Answer:

   mvn pitest:mutationCoverage -> Option B

4. Quick Check:

   PIT Maven command = mvn pitest:mutationCoverage [OK]

Hint: PIT Maven command always starts with 'mvn pitest:' [OK]

Common Mistakes:

• Using incorrect Maven goals like test:pitest
• Confusing PIT commands with other plugins
• Missing the 'mutationCoverage' goal

3. Given the following code snippet and its test, what will PIT report if the test does NOT fail after mutation?

public int add(int a, int b) { return a + b; }

Mutation changes '+' to '-'. The test asserts add(2,3) == 5.

medium

A. Mutation killed (test failed)

B. Syntax error in mutation

C. Mutation survived (test passed)

D. Test skipped due to mutation

Solution

1. Step 1: Understand mutation effect

   The '+' operator is changed to '-', so add(2,3) returns -1 instead of 5.

2. Step 2: Analyze test result

   If the test still passes, it means it did not detect the mutation, so mutation survived.

3. Final Answer:

   Mutation survived (test passed) -> Option C

4. Quick Check:

   Mutation survived = test missed bug [OK]

Hint: If test passes despite mutation, mutation survived [OK]

Common Mistakes:

• Assuming mutation killed when test passes
• Confusing mutation survival with syntax errors
• Thinking mutation causes test to skip

4. You run PIT on a JUnit project but notice no mutations are detected. Which of the following is the MOST likely cause?

medium

A. Tests are missing or not executed during mutation testing

B. PIT does not support JUnit tests

C. Mutation testing always reports zero mutations

D. Code has no executable statements

Solution

1. Step 1: Consider PIT and JUnit compatibility

   PIT supports JUnit tests, so PIT does not support JUnit tests is incorrect.

2. Step 2: Analyze why no mutations detected

   If tests are missing or not run, PIT cannot detect mutations, causing zero mutations reported.

3. Final Answer:

   Tests are missing or not executed during mutation testing -> Option A

4. Quick Check:

   No tests run = no mutations detected [OK]

Hint: No tests run means no mutation detection [OK]

Common Mistakes:

• Believing PIT doesn't support JUnit
• Thinking mutation testing always reports mutations
• Assuming code has no executable statements

5. In a JUnit project using PIT, you notice that mutation coverage is low even though your tests pass. What is the BEST way to improve mutation coverage?

hard

A. Ignore mutation coverage and focus on code coverage

B. Increase the number of mutations PIT generates

C. Disable mutation testing for complex classes

D. Add more tests that fail when code is mutated

Solution

1. Step 1: Understand mutation coverage meaning

   Low mutation coverage means tests do not fail when code is changed (mutated).

2. Step 2: Identify how to improve mutation coverage

   Adding tests that detect mutations (fail on changes) improves mutation coverage.

3. Final Answer:

   Add more tests that fail when code is mutated -> Option D

4. Quick Check:

   Better tests = higher mutation coverage [OK]

Hint: Stronger tests catch mutations, raising coverage [OK]

Common Mistakes:

• Trying to increase mutations instead of test quality
• Disabling mutation testing to hide low coverage
• Ignoring mutation coverage in favor of code coverage only