Flaky test management in Testing Fundamentals - Build an Automation Script

Detect and handle flaky test in a simple test suite

Preconditions (2)

Step 1: Run the test suite 5 times in a row without changing the code

Step 2: Record the pass or fail result of each test in each run

Step 3: Identify any test that fails in some runs but passes in others as flaky

Step 4: Mark the flaky test in the test report

Step 5: Rerun the flaky test separately 3 more times to confirm flakiness

Step 6: Log the flaky test details for further investigation

✅ Expected Result: The flaky test is detected by inconsistent pass/fail results across runs, marked clearly in the report, and rerun to confirm flakiness

Automation Requirements - pytest

Assertions Needed:

Verify test results are recorded for each run

Verify flaky test is identified by inconsistent results

Verify flaky test is marked in the report

Best Practices:

Use pytest fixtures to run tests multiple times

Use logging to record test results

Separate flaky test detection logic from test logic

Avoid hardcoding test names; use test metadata

Use clear assertion messages

Automated Solution

Testing Fundamentals

import pytest
import logging

logging.basicConfig(level=logging.INFO, format='%(message)s')

# Simulated flaky test function
import random

def flaky_test():
    # Randomly pass or fail
    return random.choice([True, False])

@pytest.fixture(scope='module')
def run_flaky_test_multiple_times():
    results = []
    for _ in range(5):
        result = flaky_test()
        results.append(result)
        logging.info(f'Run {_ + 1}: {"PASS" if result else "FAIL"}')
    return results


def test_detect_flaky(run_flaky_test_multiple_times):
    results = run_flaky_test_multiple_times
    assert len(results) == 5, "Should have 5 test run results"
    # Check if results are inconsistent
    if all(results) or not any(results):
        flaky = False
    else:
        flaky = True
    logging.info(f'Flaky detected: {flaky}')
    assert flaky, "Test should be detected as flaky due to inconsistent results"

    # Rerun flaky test 3 more times to confirm
    rerun_results = []
    for i in range(3):
        result = flaky_test()
        rerun_results.append(result)
        logging.info(f'Rerun {i + 1}: {"PASS" if result else "FAIL"}')

    # Confirm flakiness if rerun results also inconsistent
    if all(rerun_results) or not any(rerun_results):
        rerun_flaky = False
    else:
        rerun_flaky = True
    logging.info(f'Flaky confirmed on rerun: {rerun_flaky}')
    assert rerun_flaky, "Flaky test should be confirmed on rerun"

This script uses pytest to automate flaky test detection.

The flaky_test function simulates a test that randomly passes or fails.

The fixture run_flaky_test_multiple_times runs this test 5 times and logs results.

The test test_detect_flaky checks if results are inconsistent, marking the test as flaky if so.

It then reruns the flaky test 3 more times to confirm flakiness.

Assertions verify the number of runs, flaky detection, and confirmation.

Logging helps track each run's outcome for clarity.

Common Mistakes - 5 Pitfalls

Using fixed pass/fail results instead of random to simulate flakiness

Hardcoding test names or results in detection logic

Not rerunning tests to confirm flakiness

Using print statements instead of logging

Mixing flaky detection logic inside test logic

Bonus Challenge

Now add data-driven testing with 3 different flaky test functions having different flaky behaviors

Show Hint