Flaky test management in Testing Fundamentals - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Identifying Flaky Tests

Which of the following best describes a flaky test in software testing?

A A test that always fails due to a bug in the code.

B A test that is skipped during test execution.

C A test that runs slowly but always produces the same result.

D A test that sometimes passes and sometimes fails without any code changes.

💡 Hint

Think about tests that behave unpredictably without changes in the code.

A flaky test is one that produces different results (pass or fail) when run multiple times without any changes in the code or environment. This unpredictability makes it hard to trust test results.

❓ Predict Output

intermediate

Output of a Flaky Test Simulation

Consider the following Python test simulation code that randomly fails. What is the output after running it once?

Testing Fundamentals

import random

def flaky_test():
    if random.choice([True, False]):
        return 'Test Passed'
    else:
        return 'Test Failed'

result = flaky_test()
print(result)

A No output

B Test Failed

C SyntaxError

D Test Passed

💡 Hint

The output depends on random choice; consider one possible output.

The code randomly returns either 'Test Passed' or 'Test Failed'. Since the question asks for output after one run, both B and D are possible outputs.

❓ assertion

advanced

Correct Assertion to Detect Flaky Behavior

Which assertion approach best helps detect flaky tests by running the same test multiple times and checking for consistent results?

A Run the test multiple times and assert all results are the same.

B Run the test multiple times and assert at least one pass.

C Run the test once and assert the expected output.

D Run the test multiple times and assert at least one fail.

💡 Hint

Think about how to confirm a test is stable across runs.

To detect flaky tests, running the test multiple times and asserting that all results are consistent (all pass or all fail) helps identify instability. Option A ensures consistent results, which means the test is not flaky.

🔧 Debug

advanced

Debugging a Flaky Test Caused by Timing

Given this test code snippet, which change will most likely fix the flaky behavior caused by timing issues?

def test_load_data():
    data = load_data()
    assert len(data) > 0

# load_data fetches data asynchronously and may not be ready immediately

A Remove the assertion to avoid failures.

B Use a retry loop to check data readiness before asserting.

C Run the test only once to avoid flakiness.

D Add a fixed sleep delay before asserting the data length.

💡 Hint

Think about waiting for the data to be ready instead of guessing a fixed delay.

Using a retry loop to wait until the data is ready before asserting avoids flaky failures caused by timing. Fixed delays (Option B) can be unreliable and cause longer test times. Removing assertions or running once does not solve flakiness.

❓ framework

expert

Best Practice for Managing Flaky Tests in CI Pipelines

In a continuous integration (CI) pipeline, what is the best practice to handle flaky tests to maintain reliable test reports?

A Ignore flaky tests completely to avoid false failures.

B Remove flaky tests from the test suite permanently.

C Automatically retry flaky tests a fixed number of times before marking as failed.

D Run flaky tests only on developer machines, not in CI.

💡 Hint

Consider how to balance test reliability and visibility in automated pipelines.

Automatically retrying flaky tests a few times helps reduce false failures in CI reports while still keeping tests visible. Ignoring or removing flaky tests hides problems and reduces test coverage. Running only on developer machines reduces CI reliability.

Practice

1. What is a flaky test in software testing?

easy

A. A test that always passes successfully

B. A test that sometimes passes and sometimes fails without code changes

C. A test that is never executed

D. A test that always fails due to a bug

Solution

1. Step 1: Understand flaky test behavior

   A flaky test is one that shows inconsistent results, passing or failing randomly without changes in the code.

2. Step 2: Compare options to definition

   Only A test that sometimes passes and sometimes fails without code changes describes this behavior correctly; others describe consistent outcomes or unrelated cases.

3. Final Answer:

   A test that sometimes passes and sometimes fails without code changes -> Option B

4. Quick Check:

   Flaky test = random pass/fail [OK]

Hint: Flaky tests fail unpredictably without code changes [OK]

Common Mistakes:

• Thinking flaky tests always fail
• Confusing flaky tests with skipped tests
• Assuming flaky tests are due to code bugs only

2. Which of the following is the correct way to mark a flaky test in Python using pytest?

easy

A. @pytest.mark.flaky

B. @pytest.flaky()

C. @flaky_test

D. @pytest.mark.flaky(reruns=3)

Solution

1. Step 1: Recall pytest flaky test marking syntax

   Pytest uses decorators with @pytest.mark.flaky and parameters like reruns to mark flaky tests.

2. Step 2: Identify correct syntax among options

   @pytest.mark.flaky(reruns=3) correctly uses @pytest.mark.flaky(reruns=3) to rerun flaky tests up to 3 times.

3. Final Answer:

   @pytest.mark.flaky(reruns=3) -> Option D

4. Quick Check:

   pytest flaky mark = @pytest.mark.flaky(reruns=3) [OK]

Hint: Look for decorator with reruns parameter in pytest [OK]

Common Mistakes:

• Using incorrect decorator syntax
• Missing reruns parameter for flaky tests
• Confusing pytest.mark with other decorators

3. Consider this pseudocode for a flaky test management system:

tests = ['testA', 'testB', 'testC']
flaky_tests = set()

for test in tests:
    result = run_test(test)
    if result == 'fail':
        flaky_tests.add(test)

print(sorted(flaky_tests))

If run_test('testA') returns 'pass', run_test('testB') returns 'fail', and run_test('testC') returns 'fail', what will be printed?

medium

A. ['testB', 'testC']

B. ['testA', 'testB', 'testC']

C. ['testA']

D. []

Solution

1. Step 1: Analyze test results and flaky test additions

   Only tests with 'fail' results are added to flaky_tests set: testB and testC.

2. Step 2: Sort and print flaky_tests

   Sorted flaky_tests will be ['testB', 'testC'].

3. Final Answer:

   ['testB', 'testC'] -> Option A

4. Quick Check:

   Failed tests = ['testB', 'testC'] [OK]

Hint: Add only failed tests to flaky set, then sort [OK]

Common Mistakes:

• Including passed tests in flaky list
• Not sorting the output
• Confusing set behavior with list

4. You have a flaky test marked with a retry decorator, but it still fails intermittently. What is the most likely cause?

medium

A. The retry count is too low to cover all flaky failures

B. The test code has a syntax error

C. The test is not marked as flaky

D. The test always passes and never fails

Solution

1. Step 1: Understand retry decorator purpose

   Retry decorators rerun flaky tests a set number of times to reduce false failures.

2. Step 2: Identify cause of intermittent failure despite retries

   If failures persist, likely the retry count is too low to catch all flaky cases.

3. Final Answer:

   The retry count is too low to cover all flaky failures -> Option A

4. Quick Check:

   Low retry count causes flaky failures [OK]

Hint: Check retry count if flaky test still fails [OK]

Common Mistakes:

• Assuming syntax error causes flaky behavior
• Forgetting to mark test as flaky
• Ignoring retry count limits

5. A team wants to manage flaky tests without blocking releases. Which approach best balances fixing flaky tests and maintaining release flow?

hard

A. Run flaky tests only once before release

B. Remove flaky tests permanently to avoid failures

C. Temporarily mark flaky tests to skip during release, then fix them later

D. Ignore flaky tests and release as usual

Solution

1. Step 1: Understand impact of flaky tests on releases

   Flaky tests can block releases if they fail randomly, so managing them is critical.

2. Step 2: Evaluate options for balancing fixes and release flow

   Temporarily marking flaky tests to skip during release avoids blocking while allowing later fixes.

3. Final Answer:

   Temporarily mark flaky tests to skip during release, then fix them later -> Option C

4. Quick Check:

   Skip flaky tests temporarily to keep releases smooth [OK]

Hint: Skip flaky tests temporarily, fix later to keep releases smooth [OK]

Common Mistakes:

• Deleting flaky tests instead of fixing
• Ignoring flaky tests causing release failures
• Running flaky tests only once risking missed failures