Flaky test detection and retry in PyTest - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this flaky test retry code?

Consider this pytest test with a retry decorator that reruns the test up to 3 times if it fails. What will be the final test result after running this test?

PyTest

import pytest
import random

@pytest.mark.flaky(reruns=3)
def test_random_fail():
    assert random.choice([True, False])

A Test result is always skipped due to flaky marker

B Test always fails because random.choice can be False every time

C Test passes only if first run succeeds, retries ignored

D Test passes if any retry succeeds within 3 reruns

💡 Hint

The reruns parameter allows retrying the test if it fails up to the given number of times.

The @pytest.mark.flaky decorator with reruns=3 retries the test up to 3 times if it fails. If any retry passes, the test is marked as passed.

❓ assertion

intermediate

Which assertion correctly detects flaky test retry success?

You want to assert that a flaky test retried up to 2 times eventually passes. Which assertion correctly checks this in pytest?

A assert test_result.wasSuccessful()

B assert test_result.rerun_count <= 2 and test_result.passed

C assert test_result.skipped

D assert test_result.failed and test_result.rerun_count == 2

💡 Hint

Check both the number of reruns and the final pass status.

To confirm a flaky test passed after retries, ensure rerun_count is within limit and the test passed.

🔧 Debug

advanced

Why does this flaky test retry decorator not work as expected?

This pytest test uses a retry decorator but never retries on failure. Identify the bug.

PyTest

import pytest

def retry_test(func):
    def wrapper():
        for _ in range(3):
            try:
                func()
                break
            except AssertionError:
                pass
        else:
            raise AssertionError("Test failed after retries")
    return wrapper

@retry_test
def test_always_fail():
    assert False

A The wrapper does not raise AssertionError after retries, so pytest sees no failure

B The retry loop is missing a return statement after success

C The decorator is missing @pytest.mark.flaky

D The test function is missing a parameter for retries

💡 Hint

Pytest needs the test to raise AssertionError to mark failure.

The decorator swallows AssertionError and never re-raises it after retries, so pytest thinks the test passed.

🧠 Conceptual

advanced

What is the main risk of using flaky test retries without fixing root causes?

Why is relying on flaky test retries considered a bad practice in software testing?

A It increases test execution speed significantly

B It automatically fixes bugs in the code under test

C It hides real issues and reduces test reliability over time

D It guarantees all tests will eventually pass

💡 Hint

Think about test trustworthiness and maintenance.

Retries can mask flaky tests instead of fixing them, leading to unreliable test suites and missed bugs.

❓ framework

expert

Which pytest plugin and option enable automatic flaky test retries with reporting?

You want to automatically retry flaky tests up to 5 times and see retry counts in the test report. Which plugin and command line option achieve this?

A pytest-rerunfailures plugin with --reruns=5 option

B pytest-flaky plugin with --flaky-runs=5 option

C pytest-repeat plugin with --count=5 option

D pytest-xdist plugin with --max-retries=5 option

💡 Hint

Look for the plugin designed specifically for rerunning failed tests.

pytest-rerunfailures plugin supports --reruns option to retry failed tests and reports retry counts.