Flaky test detection and retry in PyTest - Build an Automation Script

Detect flaky test and retry on failure using pytest

Preconditions (2)

Step 1: Create a test function that randomly fails

Step 2: Add a decorator to retry the test up to 3 times on failure

Step 3: Run the test suite

Step 4: Observe if the test retries when it fails

Step 5: Verify the test passes if it succeeds within retries or fails after retries

✅ Expected Result: The flaky test retries up to 3 times on failure and passes if any retry succeeds; otherwise, it fails after all retries.

Automation Requirements - pytest with pytest-rerunfailures plugin

Assertions Needed:

Test passes if it succeeds within retry attempts

Test fails if all retry attempts fail

Best Practices:

Use pytest-rerunfailures plugin for retry logic

Keep flaky test logic simple and isolated

Use explicit assertions for pass/fail

Avoid hardcoding retry logic inside test functions

Automated Solution

PyTest

import random
import pytest

@pytest.mark.flaky(reruns=3, reruns_delay=1)
def test_flaky_behavior():
    # Simulate flaky behavior: 50% chance to fail
    assert random.choice([True, False]), "Random failure to simulate flakiness"

This test uses the @pytest.mark.flaky decorator from the pytest-rerunfailures plugin to retry the test up to 3 times if it fails.

The test function test_flaky_behavior randomly fails half the time by asserting a random boolean.

If the test fails, pytest automatically retries it up to 3 times with a 1-second delay between retries.

This approach cleanly separates retry logic from test logic and uses a well-supported plugin.

Common Mistakes - 3 Pitfalls

Using time.sleep inside test to wait for retry

Hardcoding retry logic inside the test function with loops

Not installing or enabling pytest-rerunfailures plugin

Bonus Challenge

Now add data-driven testing with 3 different flaky scenarios using pytest parametrize

Show Hint