Flaky test detection in JUnit

Introduction

Flaky tests sometimes pass and sometimes fail without code changes. Detecting them helps keep tests reliable and trustworthy.

When a test fails randomly on some runs but passes on others.

When you want to avoid wasting time investigating false failures.

When you notice inconsistent test results in your continuous integration.

When you want to improve confidence in your test suite.

When debugging intermittent issues caused by timing or environment.

Syntax

JUnit

Use repeated test runs or JUnit extensions to detect flaky tests.

Example with JUnit 5 @RepeatedTest:

@RepeatedTest(5)
void testSomething() {
    // test code
}

Or use a custom extension to rerun failed tests automatically.

JUnit 5 supports @RepeatedTest to run the same test multiple times.

Flaky test detection often involves running tests multiple times and checking for inconsistent results.

Examples

This test runs 3 times. It may fail sometimes if value is 8 or 9, showing flakiness.

JUnit

@RepeatedTest(3)
void testRandomBehavior() {
    int value = (int)(Math.random() * 10);
    assertTrue(value < 8);
}

This custom extension retries a failed test once (extend the logic for more retries to better detect flakiness).

JUnit

import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.jupiter.api.extension.TestExecutionExceptionHandler;
import java.lang.reflect.InvocationTargetException;

public class RetryExtension implements TestExecutionExceptionHandler {
    private int maxRetries = 3;
    private int retryCount = 0;

    @Override
    public void handleTestExecutionException(ExtensionContext context, Throwable throwable) throws Throwable {
        if (retryCount < maxRetries) {
            retryCount++;
            try {
                context.getRequiredTestMethod().invoke(context.getRequiredTestInstance());
            } catch (InvocationTargetException e) {
                handleTestExecutionException(context, e.getCause());
            }
        } else {
            throw throwable;
        }
    }
}

Sample Program

This test runs 5 times. It prints the random number each time. If the number is 8 or 9, the test fails, showing flakiness.

JUnit

import static org.junit.jupiter.api.Assertions.assertTrue;
import org.junit.jupiter.api.RepeatedTest;

public class FlakyTestExample {

    @RepeatedTest(5)
    void testRandomNumber() {
        int number = (int)(Math.random() * 10);
        System.out.println("Test run with number: " + number);
        assertTrue(number < 8, "Number is too high, test fails");
    }
}

Important Notes

Flaky tests can hide real bugs or waste time. Detecting them early is important.

Running tests multiple times helps reveal flaky behavior caused by timing or randomness.

Use logging or print statements to see test behavior during repeated runs.

Summary

Flaky tests sometimes pass and sometimes fail without code changes.

Detect flaky tests by running them multiple times or using retry extensions.

Fix flaky tests to keep your test suite reliable and trustworthy.