Flaky test detection and retry in PyTest - Framework Patterns

Framework Mode - Flaky test detection and retry

Folder Structure

tests/
├── test_login.py
├── test_checkout.py
├── test_profile.py
conftest.py
pytest.ini
utils/
├── retry_helper.py
├── flaky_detector.py
reports/
└── latest_report.html

Test Framework Layers

• Tests: Individual test files inside tests/ folder, e.g., test_login.py. These contain test functions using pytest.
• Fixtures & Hooks: Defined in conftest.py for setup, teardown, and retry hooks.
• Utilities: Helper modules like retry_helper.py to implement retry logic and flaky_detector.py to log flaky test info.
• Configuration: pytest.ini to configure pytest plugins and retry options.
• Reports: Generated test reports stored in reports/ folder for review.

Configuration Patterns

• pytest.ini: Configure retry plugin with max retries and delay, e.g.,

  [pytest]
  addopts = --reruns 2 --reruns-delay 1
  

  This retries failed tests up to 2 times with 1 second delay.
• Environment Variables: Use environment variables or .env files to toggle flaky detection on/off without code changes.
• Custom Hooks: Use pytest hooks in conftest.py to detect flaky tests by tracking retries and logging flaky occurrences.

Test Reporting and CI/CD Integration

• Use pytest-html or Allure plugins to generate detailed HTML reports showing retry attempts and flaky test status.
• Reports saved in reports/ folder for easy access and historical comparison.
• Integrate test runs with CI/CD pipelines (e.g., GitHub Actions, Jenkins) to automatically run tests with retry enabled and fail builds on persistent failures.
• Use report annotations or comments in CI to highlight flaky tests for team review.

Best Practices

1. Limit retries: Retry only a small number of times (e.g., 2-3) to avoid masking real issues.
2. Log flaky tests: Keep track of tests that pass only after retries to identify unstable tests.
3. Use explicit waits: Combine retries with proper waits to reduce flakiness caused by timing issues.
4. Isolate flaky tests: Mark flaky tests with custom markers to run them separately if needed.
5. Review flaky tests regularly: Fix flaky tests promptly to maintain test suite reliability.

Self Check

Where in this folder structure would you add a new utility function to log flaky test occurrences during retries?

Key Result

Use pytest retry plugin with custom hooks and utilities to detect and handle flaky tests effectively.