Flaky test management in Testing Fundamentals - Framework Patterns

Framework Mode - Flaky test management

Folder Structure for Flaky Test Management

flaky-test-project/
├── tests/
│   ├── stable/
│   │   └── test_login.py
│   ├── flaky/
│   │   └── test_payment_retry.py
│   └── __init__.py
├── utils/
│   ├── flaky_handler.py
│   ├── retry_decorator.py
│   └── __init__.py
├── config/
│   ├── environments.yaml
│   └── settings.yaml
├── reports/
│   └── latest_report.html
├── conftest.py
└── pytest.ini

Test Framework Layers for Flaky Test Management

• Test Layer: Contains test cases separated into stable and flaky folders to isolate flaky tests.
• Utility Layer: Includes helpers like retry_decorator.py to rerun flaky tests automatically and flaky_handler.py to mark or log flaky tests.
• Configuration Layer: Holds environment and test settings to control test runs and retries.
• Reporting Layer: Generates test reports highlighting flaky test results for easy review.
• Test Runner Layer: Uses pytest with plugins or custom hooks to manage flaky test retries and reporting.

Configuration Patterns for Flaky Test Management

• Retry Settings: Define retry count and delay in pytest.ini or settings.yaml to control how many times flaky tests rerun before failing.
• Environment Variables: Use environment flags to enable or disable flaky test retries during CI or local runs.
• Marking Flaky Tests: Use decorators or markers (e.g., @pytest.mark.flaky) to identify flaky tests clearly.
• Logging: Configure detailed logs for flaky test runs to analyze failure patterns.

Test Reporting and CI/CD Integration

• Flaky Test Reports: Generate separate reports or highlight flaky tests in main reports to track their stability over time.
• CI/CD Integration: Configure pipelines to rerun flaky tests automatically and fail builds only after retries are exhausted.
• Notifications: Send alerts for flaky test failures to the team for investigation.
• Trend Analysis: Store historical flaky test data to identify patterns and improve test reliability.

Best Practices for Flaky Test Management

1. Isolate Flaky Tests: Keep flaky tests separate from stable ones to avoid blocking the whole test suite.
2. Use Automatic Retries: Implement retries with limits to reduce false failures without hiding real issues.
3. Mark and Track: Clearly mark flaky tests and track their history to prioritize fixing them.
4. Analyze Root Causes: Regularly review flaky test failures to find and fix underlying problems.
5. Keep Tests Simple and Reliable: Design tests to minimize external dependencies and timing issues that cause flakiness.

Self Check Question

Where in this folder structure would you add a new utility to log flaky test retries?

Key Result

Separate flaky tests, use retries, mark clearly, and track to manage flaky test reliability.