Flaky test detection in JUnit - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is a flaky test?

In software testing, what best describes a flaky test?

A A test that sometimes passes and sometimes fails without code changes

B A test that always passes regardless of code changes

C A test that never runs due to configuration errors

D A test that only runs on weekends

💡 Hint

Think about tests that behave unpredictably.

A flaky test is one that produces different results (pass or fail) without any changes in the code, often due to timing, environment, or dependencies.

❓ Predict Output

intermediate

JUnit test with flaky behavior output

What will be the output of this JUnit test if it is flaky due to timing?

JUnit

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertTrue;

public class FlakyTest {
    @Test
    void testRandomPass() throws InterruptedException {
        Thread.sleep((long)(Math.random() * 100));
        assertTrue(System.currentTimeMillis() % 2 == 0);
    }
}

A Test sometimes passes and sometimes fails

B Test fails every time

C Test passes every time

D Compilation error due to Thread.sleep

💡 Hint

Look at the assertion and the use of random timing.

The test assertion depends on the current time's evenness, which changes every millisecond, causing the test to pass or fail unpredictably.

❓ assertion

advanced

Identify the assertion that detects flaky test behavior

Which assertion best helps detect flaky tests by running the same test multiple times?

A assertEquals(expected, actual)

B assertTrue(runTestMultipleTimes() >= 1)

C assertDoesNotThrow(() -> method())

D assertTimeout(Duration.ofSeconds(1), () -> method())

💡 Hint

Consider how to check if a test passes at least once in multiple runs.

Running a test multiple times and asserting it passes at least once helps detect flaky behavior.

🔧 Debug

advanced

Debugging a flaky test caused by shared state

Given this JUnit test code, what is the most likely cause of flakiness?

JUnit

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertTrue;

public class SharedStateTest {
    private static int counter = 0;

    @Test
    void testIncrement() {
        counter++;
        assertTrue(counter <= 1);
    }
}

A assertTrue condition is always true

B counter variable is local and resets each test

C Test method is missing @BeforeEach annotation

D counter is not reset between tests causing shared state issues

💡 Hint

Think about static variables and test isolation.

The static counter variable keeps its value between test runs, causing tests to fail when run multiple times or in parallel.

❓ framework

expert

Using JUnit 5 to detect flaky tests automatically

Which JUnit 5 feature helps detect flaky tests by repeating a test multiple times and reporting failures?

A @BeforeAll annotation for setup before all tests

B @Test annotation with timeout parameter

C @RepeatedTest annotation with a fixed number of repetitions

D @Disabled annotation to skip flaky tests

💡 Hint

Look for annotations that run tests multiple times.

The @RepeatedTest annotation runs the same test multiple times, helping identify flaky tests by observing inconsistent results.