Mutation testing concept (PIT) in JUnit - Framework Patterns

Framework Mode - Mutation testing concept (PIT)

Folder Structure for Mutation Testing with PIT and JUnit

project-root/
├── src/
│   ├── main/
│   │   └── java/
│   │       └── com/example/app/
│   │           └── Calculator.java
│   └── test/
│       └── java/
│           └── com/example/app/
│               └── CalculatorTest.java
├── pom.xml  (or build.gradle)
├── pitest.properties
└── reports/
    └── pit-reports/
  

Test Framework Layers for Mutation Testing with PIT

• Production Code Layer: Contains the application code to be tested (e.g., Calculator.java).
• Test Layer: JUnit test classes that verify the production code behavior (e.g., CalculatorTest.java).
• Mutation Testing Layer: PIT tool integrates with the build to mutate production code and run tests to check if mutations are caught.
• Configuration Layer: PIT configuration files or build tool settings to specify mutation settings, target classes, and reports.
• Reporting Layer: Generates mutation testing reports showing killed and survived mutants.

Configuration Patterns for PIT Mutation Testing

Use pitest.properties or build tool plugins (Maven/Gradle) to configure:

• Target Classes: Specify which classes PIT should mutate (e.g., com.example.app.*).
• Test Classes: Specify test classes to run against mutants.
• Mutation Operators: Choose mutation types (e.g., negate conditionals, replace math operators).
• Timeouts and Threads: Control test execution time and parallelism.
• Report Directory: Define where mutation reports are saved.

Example Maven plugin snippet in pom.xml:

<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <version>1.9.11</version>
  <configuration>
    <targetClasses>
      <param>com.example.app.*</param>
    </targetClasses>
    <targetTests>
      <param>com.example.app.*Test</param>
    </targetTests>
    <mutationOperators>
      <param>DEFAULTS</param>
    </mutationOperators>
    <outputFormats>
      <param>HTML</param>
    </outputFormats>
  </configuration>
</plugin>
  

Test Reporting and CI/CD Integration

PIT generates detailed HTML reports showing:

• Killed Mutants: Mutations detected by tests (good).
• Survived Mutants: Mutations not detected (need better tests).
• Mutation Coverage: Percentage of mutants killed.

Reports are saved in reports/pit-reports/ folder.

Integrate PIT in CI/CD pipelines (Jenkins, GitHub Actions) by running mutation tests as part of build. Fail build if mutation coverage is below threshold.

Best Practices for Mutation Testing Framework with PIT

• Keep Tests Fast: Mutation testing runs many tests; optimize test speed.
• Use Clear Configuration: Explicitly define target classes and tests to avoid unnecessary mutations.
• Analyze Survived Mutants: Use reports to improve test cases where mutants survive.
• Integrate with CI/CD: Automate mutation testing to maintain high test quality over time.
• Run Mutation Testing Periodically: Because it is resource-intensive, run mutation tests nightly or on demand.

Self-Check Question

Where in this folder structure would you add a new JUnit test class to improve mutation coverage for the Calculator class?

Key Result

Mutation testing with PIT uses JUnit tests to check if small code changes (mutations) are caught, improving test quality.

Practice

1. What is the main purpose of mutation testing using PIT in JUnit projects?

easy

A. To check if tests can detect small changes (mutations) in the code

B. To improve the performance of the application

C. To generate new test cases automatically

D. To refactor the code for better readability

Solution

1. Step 1: Understand mutation testing concept

   Mutation testing changes code slightly to see if tests catch the changes.

2. Step 2: Identify PIT's role in JUnit projects

   PIT automates mutation testing to check test effectiveness.

3. Final Answer:

   To check if tests can detect small changes (mutations) in the code -> Option A

4. Quick Check:

   Mutation testing = detect code changes [OK]

Hint: Mutation testing checks test strength by code changes [OK]

Common Mistakes:

• Thinking mutation testing improves app speed
• Confusing mutation testing with test generation
• Assuming mutation testing refactors code

2. Which of the following is the correct way to run PIT mutation testing with Maven in a JUnit project?

easy

A. mvn test:pitest

B. mvn pitest:mutationCoverage

C. mvn mutation:test

D. mvn pit:run

Solution

1. Step 1: Recall PIT Maven command

   The standard Maven command to run PIT is mvn pitest:mutationCoverage.

2. Step 2: Check other options for correctness

   Other options are invalid or do not match PIT's official command syntax.

3. Final Answer:

   mvn pitest:mutationCoverage -> Option B

4. Quick Check:

   PIT Maven command = mvn pitest:mutationCoverage [OK]

Hint: PIT Maven command always starts with 'mvn pitest:' [OK]

Common Mistakes:

• Using incorrect Maven goals like test:pitest
• Confusing PIT commands with other plugins
• Missing the 'mutationCoverage' goal

3. Given the following code snippet and its test, what will PIT report if the test does NOT fail after mutation?

public int add(int a, int b) { return a + b; }

Mutation changes '+' to '-'. The test asserts add(2,3) == 5.

medium

A. Mutation killed (test failed)

B. Syntax error in mutation

C. Mutation survived (test passed)

D. Test skipped due to mutation

Solution

1. Step 1: Understand mutation effect

   The '+' operator is changed to '-', so add(2,3) returns -1 instead of 5.

2. Step 2: Analyze test result

   If the test still passes, it means it did not detect the mutation, so mutation survived.

3. Final Answer:

   Mutation survived (test passed) -> Option C

4. Quick Check:

   Mutation survived = test missed bug [OK]

Hint: If test passes despite mutation, mutation survived [OK]

Common Mistakes:

• Assuming mutation killed when test passes
• Confusing mutation survival with syntax errors
• Thinking mutation causes test to skip

4. You run PIT on a JUnit project but notice no mutations are detected. Which of the following is the MOST likely cause?

medium

A. Tests are missing or not executed during mutation testing

B. PIT does not support JUnit tests

C. Mutation testing always reports zero mutations

D. Code has no executable statements

Solution

1. Step 1: Consider PIT and JUnit compatibility

   PIT supports JUnit tests, so PIT does not support JUnit tests is incorrect.

2. Step 2: Analyze why no mutations detected

   If tests are missing or not run, PIT cannot detect mutations, causing zero mutations reported.

3. Final Answer:

   Tests are missing or not executed during mutation testing -> Option A

4. Quick Check:

   No tests run = no mutations detected [OK]

Hint: No tests run means no mutation detection [OK]

Common Mistakes:

• Believing PIT doesn't support JUnit
• Thinking mutation testing always reports mutations
• Assuming code has no executable statements

5. In a JUnit project using PIT, you notice that mutation coverage is low even though your tests pass. What is the BEST way to improve mutation coverage?

hard

A. Ignore mutation coverage and focus on code coverage

B. Increase the number of mutations PIT generates

C. Disable mutation testing for complex classes

D. Add more tests that fail when code is mutated

Solution

1. Step 1: Understand mutation coverage meaning

   Low mutation coverage means tests do not fail when code is changed (mutated).

2. Step 2: Identify how to improve mutation coverage

   Adding tests that detect mutations (fail on changes) improves mutation coverage.

3. Final Answer:

   Add more tests that fail when code is mutated -> Option D

4. Quick Check:

   Better tests = higher mutation coverage [OK]

Hint: Stronger tests catch mutations, raising coverage [OK]

Common Mistakes:

• Trying to increase mutations instead of test quality
• Disabling mutation testing to hide low coverage
• Ignoring mutation coverage in favor of code coverage only