Flaky test detection and retry in PyTest

Introduction

Sometimes tests fail randomly even if the code is correct. Detecting and retrying these flaky tests helps keep your test results reliable.

When a test fails sometimes but passes other times without code changes.

When network or timing issues cause occasional test failures.

When tests depend on external services that may be slow or unstable.

When you want to avoid false alarms from random test failures.

When you want to automatically rerun failed tests to confirm if they are truly broken.

Syntax

PyTest

import pytest

@pytest.mark.flaky(reruns=number_of_retries)
def test_example():
    # test code here

The @pytest.mark.flaky decorator tells pytest to rerun the test if it fails.

Set reruns to the number of times you want to retry the test.

Examples

This test may fail randomly. It will retry up to 2 times if it fails.

PyTest

import pytest

@pytest.mark.flaky(reruns=2)
def test_random_fail():
    import random
    assert random.choice([True, False])

This test retries 3 times with 1 second delay between retries.

PyTest

import pytest

@pytest.mark.flaky(reruns=3, reruns_delay=1)
def test_with_delay():
    assert False  # always fails

Sample Program

This test sometimes fails because it randomly chooses True or False. Pytest will retry it up to 3 times if it fails.

PyTest

import pytest
import random

@pytest.mark.flaky(reruns=3)
def test_flaky():
    # This test randomly fails
    assert random.choice([True, False])

if __name__ == "__main__":
    pytest.main(["-v", __file__])

Important Notes

Use retries sparingly to avoid hiding real problems.

Flaky tests should be fixed eventually, not just retried.

Pytest-flaky plugin or built-in rerunfailures plugin can be used for retries.

Summary

Flaky tests fail randomly and need special handling.

Use @pytest.mark.flaky(reruns=N) to retry tests automatically.

Retries help reduce false failures but don't replace fixing flaky tests.