Flaky test detection and retry in PyTest - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this flaky test retry code?

Consider this pytest test with a retry decorator that reruns the test up to 3 times if it fails. What will be the final test result after running this test?

PyTest

import pytest
import random

@pytest.mark.flaky(reruns=3)
def test_random_fail():
    assert random.choice([True, False])

A Test result is always skipped due to flaky marker

B Test always fails because random.choice can be False every time

C Test passes only if first run succeeds, retries ignored

D Test passes if any retry succeeds within 3 reruns

💡 Hint

The reruns parameter allows retrying the test if it fails up to the given number of times.

The @pytest.mark.flaky decorator with reruns=3 retries the test up to 3 times if it fails. If any retry passes, the test is marked as passed.

❓ assertion

intermediate

Which assertion correctly detects flaky test retry success?

You want to assert that a flaky test retried up to 2 times eventually passes. Which assertion correctly checks this in pytest?

A assert test_result.wasSuccessful()

B assert test_result.rerun_count <= 2 and test_result.passed

C assert test_result.skipped

D assert test_result.failed and test_result.rerun_count == 2

💡 Hint

Check both the number of reruns and the final pass status.

To confirm a flaky test passed after retries, ensure rerun_count is within limit and the test passed.

🔧 Debug

advanced

Why does this flaky test retry decorator not work as expected?

This pytest test uses a retry decorator but never retries on failure. Identify the bug.

PyTest

import pytest

def retry_test(func):
    def wrapper():
        for _ in range(3):
            try:
                func()
                break
            except AssertionError:
                pass
        else:
            raise AssertionError("Test failed after retries")
    return wrapper

@retry_test
def test_always_fail():
    assert False

A The wrapper does not raise AssertionError after retries, so pytest sees no failure

B The retry loop is missing a return statement after success

C The decorator is missing @pytest.mark.flaky

D The test function is missing a parameter for retries

💡 Hint

Pytest needs the test to raise AssertionError to mark failure.

The decorator swallows AssertionError and never re-raises it after retries, so pytest thinks the test passed.

🧠 Conceptual

advanced

What is the main risk of using flaky test retries without fixing root causes?

Why is relying on flaky test retries considered a bad practice in software testing?

A It increases test execution speed significantly

B It automatically fixes bugs in the code under test

C It hides real issues and reduces test reliability over time

D It guarantees all tests will eventually pass

💡 Hint

Think about test trustworthiness and maintenance.

Retries can mask flaky tests instead of fixing them, leading to unreliable test suites and missed bugs.

❓ framework

expert

Which pytest plugin and option enable automatic flaky test retries with reporting?

You want to automatically retry flaky tests up to 5 times and see retry counts in the test report. Which plugin and command line option achieve this?

A pytest-rerunfailures plugin with --reruns=5 option

B pytest-flaky plugin with --flaky-runs=5 option

C pytest-repeat plugin with --count=5 option

D pytest-xdist plugin with --max-retries=5 option

💡 Hint

Look for the plugin designed specifically for rerunning failed tests.

pytest-rerunfailures plugin supports --reruns option to retry failed tests and reports retry counts.

Practice

1. What is the main purpose of marking a test as flaky in pytest using @pytest.mark.flaky(reruns=N)?

easy

A. To skip the test permanently

B. To automatically retry the test N times if it fails

C. To mark the test as slow and run it last

D. To run the test only once without retries

Solution

1. Step 1: Understand the flaky test concept

   Flaky tests fail randomly due to timing or environment issues, so retries help reduce false failures.

2. Step 2: Analyze the effect of @pytest.mark.flaky(reruns=N)

   This decorator tells pytest to rerun the test up to N times if it fails, increasing chances of passing despite flakiness.

3. Final Answer:

   To automatically retry the test N times if it fails -> Option B

4. Quick Check:

   Flaky test retry = reruns N times [OK]

Hint: Retries mean automatic rerun on failure [OK]

Common Mistakes:

• Confusing flaky with skipped tests
• Thinking flaky marks slow tests
• Assuming flaky disables retries

2. Which of the following is the correct syntax to retry a flaky test 3 times in pytest?

easy

A. @pytest.flaky(tries=3)

B. @pytest.retry(3)

C. @pytest.mark.retry(3)

D. @pytest.mark.flaky(reruns=3)

Solution

1. Step 1: Recall the correct decorator name and parameter

   The correct decorator is @pytest.mark.flaky with parameter reruns to specify retry count.

2. Step 2: Match syntax with options

   @pytest.mark.flaky(reruns=3) uses @pytest.mark.flaky(reruns=3), which is the correct syntax for retrying 3 times.

3. Final Answer:

   @pytest.mark.flaky(reruns=3) -> Option D

4. Quick Check:

   Decorator name + reruns param = correct syntax [OK]

Hint: Use @pytest.mark.flaky with reruns param [OK]

Common Mistakes:

• Using @pytest.retry instead of @pytest.mark.flaky
• Using wrong parameter names like tries
• Missing the mark prefix

3. Given this pytest test code snippet, what will be the output if the test randomly fails once and passes on retry?

import pytest

@pytest.mark.flaky(reruns=2)
def test_random():
    import random
    assert random.choice([True, False])

medium

A. Test may pass after 1 or 2 retries if random returns True

B. Test always fails because random.choice is unpredictable

C. Test runs only once and fails if random returns False

D. Test is skipped due to flaky mark

Solution

1. Step 1: Understand the flaky decorator effect

   The test will rerun up to 2 times if it fails, allowing multiple chances to pass.

2. Step 2: Analyze the random assertion

   The assertion randomly passes or fails. If it fails once, retry can pass if random returns True on retry.

3. Final Answer:

   Test may pass after 1 or 2 retries if random returns True -> Option A

4. Quick Check:

   Retries allow passing despite random failure [OK]

Hint: Retries give multiple chances to pass random failures [OK]

Common Mistakes:

• Assuming flaky skips tests
• Thinking test always fails on first try
• Ignoring retry count effect

4. You wrote this flaky test with retry but it never retries on failure. What is the error?

import pytest

@pytest.mark.flaky(rerun=3)
def test_example():
    assert False

medium

A. The test must return True to retry

B. The decorator should be @pytest.retry, not @pytest.mark.flaky

C. The parameter name should be 'reruns', not 'rerun'

D. The test cannot retry if it always fails

Solution

1. Step 1: Check the decorator parameter spelling

   The correct parameter for retry count is reruns, not rerun.

2. Step 2: Understand impact of wrong parameter

   Using rerun is ignored by pytest, so no retries happen despite failures.

3. Final Answer:

   The parameter name should be 'reruns', not 'rerun' -> Option C

4. Quick Check:

   Correct param spelling = reruns [OK]

Hint: Check exact parameter spelling: reruns, not rerun [OK]

Common Mistakes:

• Misspelling reruns parameter
• Using wrong decorator name
• Expecting retries on always failing test

5. You want to reduce false failures from a flaky test that sometimes fails due to timing issues. Which approach best combines flaky test detection and retry in pytest?

hard

A. Use @pytest.mark.flaky(reruns=3) and add explicit wait in test code

B. Use @pytest.mark.skip to ignore the flaky test

C. Remove retries and fix test to always pass

D. Use @pytest.mark.flaky(reruns=0) to disable retries

Solution

1. Step 1: Understand flaky test retry purpose

   Retries help reduce false failures by rerunning tests that fail randomly.

2. Step 2: Combine retry with test stabilization

   Adding explicit waits addresses timing issues, improving test stability alongside retries.

3. Step 3: Evaluate other options

   Skipping ignores tests, removing retries loses retry benefit, and reruns=0 disables retries.

4. Final Answer:

   Use @pytest.mark.flaky(reruns=3) and add explicit wait in test code -> Option A

5. Quick Check:

   Retries + waits = best flaky test handling [OK]

Hint: Combine retries with waits to fix flaky timing issues [OK]

Common Mistakes:

• Skipping flaky tests instead of fixing
• Disabling retries by setting reruns=0
• Ignoring timing issues causing flakiness