Flaky test detection in JUnit - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to mark a test as flaky using JUnit 5.

JUnit

@Test
@[1](5)
void testRandomBehavior() {
    // test code here
}

Drag options to blanks, or click blank then click option'

AFlakyTest

BRepeatedTest

CTimeout

DDisabled

Attempts:

3 left

💡 Hint

Common Mistakes

Using @FlakyTest which does not exist in JUnit 5.

Using @Timeout which controls time limits, not repetitions.

JUnit 5 uses @RepeatedTest to run a test multiple times to detect flaky behavior.

2fill in blank

medium

Complete the code to assert that a flaky test passes at least once in multiple runs.

JUnit

int passCount = 0;
for (int i = 0; i < 10; i++) {
    try {
        flakyTest();
        passCount[1];
    } catch (Exception e) {
        // ignore failure
    }
}
assertTrue(passCount > 0);

Drag options to blanks, or click blank then click option'

A++

B--

C+=

D=

Attempts:

3 left

💡 Hint

Common Mistakes

Using '--' which decreases the count.

Using '=' which assigns instead of increments.

Using '++' increments passCount by one each time the test passes.

3fill in blank

hard

Fix the error in the flaky test retry logic by completing the blank.

JUnit

int retries = 3;
while (retries > 0) {
    try {
        flakyTest();
        break;
    } catch (Exception e) {
        retries[1];
        if (retries == 0) throw e;
    }
}

Drag options to blanks, or click blank then click option'

A=

B++

C+=

D--

Attempts:

3 left

💡 Hint

Common Mistakes

Using '++' which increases retries causing infinite loop.

Using '=' which assigns instead of decrementing.

Decrement retries with '--' to limit retry attempts.

4fill in blank

hard

Fill both blanks to create a JUnit 5 extension that retries flaky tests up to 3 times.

JUnit

public class RetryExtension implements TestExecutionExceptionHandler {
    private int maxRetries = [1];
    private int retryCount = 0;

    @Override
    public void handleTestExecutionException(ExtensionContext context, Throwable throwable) throws Throwable {
        if (retryCount < maxRetries) {
            retryCount[2];
            context.getRequiredTestMethod().invoke(context.getRequiredTestInstance());
        } else {
            throw throwable;
        }
    }
}

Drag options to blanks, or click blank then click option'

A3

B1

C++

D--

Attempts:

3 left

💡 Hint

Common Mistakes

Setting maxRetries to 1 which allows only one attempt.

Decrementing retryCount which causes logic errors.

maxRetries is set to 3 and retryCount is incremented with '++' on each retry.

5fill in blank

hard

Fill all three blanks to implement a flaky test detection method that runs a test 5 times and returns true if it passes at least once.

JUnit

public boolean isFlaky(Runnable test) {
    int passCount = 0;
    for (int i = 0; i < [1]; i++) {
        try {
            test.run();
            passCount [2];
        } catch (Exception e) {
            // ignore failure
        }
    }
    return passCount [3] 0;
}

Drag options to blanks, or click blank then click option'

A5

B++

C>

D<

Attempts:

3 left

💡 Hint

Common Mistakes

Using less than '<' instead of greater than '>' in the return statement.

Not incrementing passCount correctly.

Run test 5 times, increment passCount with '++', and check if passCount is greater than 0.

Practice

1.

What is a flaky test in JUnit?

easy

A. A test that runs only once

B. A test that sometimes passes and sometimes fails without code changes

C. A test that always fails

D. A test that always passes

Solution

1. Step 1: Understand flaky test behavior

   A flaky test is one that does not have consistent results; it may pass or fail unpredictably even if the code hasn't changed.

2. Step 2: Identify the correct definition

   Among the options, only D describes this behavior accurately.

3. Final Answer:

   A test that sometimes passes and sometimes fails without code changes -> Option B

4. Quick Check:

   Flaky test = inconsistent pass/fail [OK]

Hint: Flaky tests fail unpredictably without code changes [OK]

Common Mistakes:

• Thinking flaky tests always fail
• Confusing flaky tests with slow tests
• Assuming flaky tests run only once

2.

Which JUnit feature helps detect flaky tests by rerunning failed tests automatically?

easy

A. JUnit RetryRule or @RepeatedTest

B. JUnit @BeforeAll annotation

C. JUnit @Test timeout parameter

D. JUnit @Ignore annotation

Solution

1. Step 1: Identify rerun features in JUnit

   JUnit provides RetryRule or can use @RepeatedTest to rerun tests multiple times to detect flaky behavior.

2. Step 2: Eliminate unrelated options

   @BeforeAll runs setup once, @Test timeout limits test time, and @Ignore skips tests, so they don't detect flaky tests.

3. Final Answer:

   JUnit RetryRule or @RepeatedTest -> Option A

4. Quick Check:

   RetryRule reruns tests to detect flakes [OK]

Hint: RetryRule reruns failed tests automatically [OK]

Common Mistakes:

• Confusing @BeforeAll with retry logic
• Using @Ignore to detect flaky tests
• Thinking timeout detects flakiness

3.

Consider this JUnit test snippet using a retry rule to detect flaky tests:

public class FlakyTest {
  private int runCount = 0;

  @Rule
  public RetryRule retryRule = new RetryRule(3);

  @Test
  public void testFlaky() {
    runCount++;
    assertTrue(runCount >= 2);
  }
}

What will be the test result when run?

medium

A. Test fails after all retries

B. Test fails immediately

C. Test passes on first try

D. Test passes after retrying once

Solution

1. Step 1: Analyze runCount increment and assertion

   On first run, runCount is 1, so assertTrue(runCount >= 2) fails. RetryRule reruns test up to 3 times.

2. Step 2: Check retry behavior

   On second run, runCount is 2, assertion passes, so test passes after one retry.

3. Final Answer:

   Test passes after retrying once -> Option D

4. Quick Check:

   Retry allows flaky test to pass on retry [OK]

Hint: Check runCount and retries to predict pass/fail [OK]

Common Mistakes:

• Assuming test passes on first try
• Ignoring retry rule effect
• Thinking test fails after all retries

4.

Identify the problem in this JUnit test intended to detect flaky tests by retrying:

@Rule
public RetryRule retryRule = new RetryRule(3);

@Test
public void testFlaky() {
  int count = 0;
  count++;
  assertTrue(count > 1);
}

medium

A. assertTrue condition is always true

B. RetryRule is not applied correctly without @Retry annotation

C. The count variable resets on each test run, so retry never passes

D. Test method lacks @Before annotation

Solution

1. Step 1: Analyze count variable scope

   Count is a local variable inside the test method, so it resets to 0 on every retry.

2. Step 2: Understand retry effect on count

   Since count resets each time, count++ makes it 1, so assertTrue(count > 1) always fails, retry won't help.

3. Final Answer:

   The count variable resets on each test run, so retry never passes -> Option C

4. Quick Check:

   Local variables reset each retry causing failure [OK]

Hint: Check variable scope to ensure state persists across retries [OK]

Common Mistakes:

• Assuming retry fixes all failures
• Ignoring variable reset on retries
• Confusing retry annotations

5.

You want to detect flaky tests in your JUnit suite by running each test 5 times and marking it flaky if it fails at least once. Which approach is best?

hard

A. Use @RepeatedTest(5) and record failures to identify flaky tests

B. Use @Ignore on all tests and run manually

C. Use @BeforeEach to reset test state only

D. Use @Timeout to fail slow tests

Solution

1. Step 1: Understand flaky test detection by multiple runs

   Running tests multiple times helps catch intermittent failures indicating flakiness.

2. Step 2: Choose correct JUnit feature

   @RepeatedTest(5) runs the test 5 times; by recording failures during these runs, flaky tests can be detected.

3. Final Answer:

   Use @RepeatedTest(5) and record failures to identify flaky tests -> Option A

4. Quick Check:

   Multiple runs with @RepeatedTest detect flaky tests [OK]

Hint: Run tests multiple times with @RepeatedTest to find flakes [OK]

Common Mistakes:

• Using @Ignore disables tests instead of detecting flakes
• Relying on @Timeout for flaky detection
• Resetting state without multiple runs