Flaky test detection in JUnit - Build an Automation Script

Detect flaky test by running a test multiple times

Preconditions (2)

Step 1: Create a test method that runs the flaky test 5 times

Step 2: In each run, execute the flaky test logic

Step 3: Record if the test passes or fails each time

Step 4: After all runs, check if there were both passes and failures

Step 5: If yes, mark the test as flaky

✅ Expected Result: The test detects if the flaky test passes sometimes and fails sometimes, indicating flakiness

Automation Requirements - JUnit 5

Assertions Needed:

Assert that the flaky test passes at least once

Assert that the flaky test fails at least once to confirm flakiness

Best Practices:

Use @RepeatedTest or loop inside test to run multiple times

Use assertions to track pass/fail counts

Keep flaky test logic isolated for clarity

Avoid hardcoding sleep or waits

Use descriptive test method names

Automated Solution

JUnit

import static org.junit.jupiter.api.Assertions.*;
import org.junit.jupiter.api.Test;

public class FlakyTestDetection {

    // Simulated flaky test method
    boolean flakyTest() {
        // Randomly pass or fail
        return Math.random() > 0.5;
    }

    @Test
    void detectFlakyTest() {
        int passCount = 0;
        int failCount = 0;
        int runs = 5;

        for (int i = 0; i < runs; i++) {
            if (flakyTest()) {
                passCount++;
            } else {
                failCount++;
            }
        }

        assertTrue(passCount > 0, "Test should pass at least once");
        assertTrue(failCount > 0, "Test should fail at least once to confirm flakiness");
    }
}

This code defines a flakyTest() method that randomly passes or fails to simulate flakiness.

The detectFlakyTest() method runs this flaky test 5 times in a loop.

It counts how many times the test passes and fails.

Then it asserts that the test passed at least once and failed at least once.

This confirms the test is flaky because it does not always behave the same.

Using a loop inside a single test method keeps the code simple and clear.

Common Mistakes - 3 Pitfalls

{'mistake': 'Running flaky test only once', 'why_bad': "You cannot detect flakiness if you run the test a single time because you won't see intermittent failures.", 'correct_approach': 'Run the test multiple times in a loop or use @RepeatedTest to observe different outcomes.'}

Using Thread.sleep to fix flakiness

Not asserting both pass and fail outcomes

Bonus Challenge

Now add data-driven testing with 3 different flaky test scenarios using @ParameterizedTest

Show Hint