Deterministic tests in PyTest - Framework Patterns

Framework Mode - Deterministic tests

Folder Structure

tests/
├── test_example.py
├── test_utils.py
├── data/
│   └── test_data.json
├── fixtures/
│   └── conftest.py
utils/
├── helpers.py
configs/
├── config.yaml
pytest.ini

Test Framework Layers

• Tests: Actual test files inside tests/ folder, e.g., test_example.py. These contain test functions using pytest.
• Fixtures: Setup and teardown code in tests/fixtures/conftest.py to prepare test environment and data.
• Test Data: Static data files in tests/data/ used to provide consistent inputs for tests.
• Utilities: Helper functions in utils/helpers.py to support tests without side effects.
• Configuration: Settings like environment variables, browser options, and credentials stored in configs/config.yaml and pytest.ini.

Configuration Patterns

• Environment Config: Use configs/config.yaml to define environments (dev, test, prod) with fixed URLs and credentials.
• pytest.ini: Configure pytest options like markers and test paths to keep tests organized and consistent.
• Fixtures for Setup: Use conftest.py fixtures to initialize resources deterministically before tests run.
• Seed Data: Load fixed test data from JSON files to ensure tests always get the same inputs.
• Avoid Randomness: Do not use random or time-dependent data unless seeded and controlled.

Test Reporting and CI/CD Integration

• pytest Reports: Use built-in pytest reports with --junitxml=report.xml for CI systems.
• CI Integration: Configure CI pipelines (GitHub Actions, Jenkins) to run tests on every commit ensuring deterministic results.
• Fail Fast: Enable pytest options to stop on first failure to quickly detect flaky or non-deterministic tests.
• Logs and Artifacts: Save logs and screenshots on failure to help diagnose non-deterministic behavior.

Best Practices for Deterministic Tests

1. Fixed Test Data: Always use static or seeded data to avoid randomness.
2. Isolate Tests: Each test should run independently without relying on others.
3. Use Fixtures: Setup and teardown with fixtures to ensure consistent environment state.
4. Avoid External Dependencies: Mock or stub external services to prevent variability.
5. Control Time: Use time freezing or fixed timestamps if tests depend on time.

Self Check

Where would you add a new fixture that sets up a database connection with fixed test data for deterministic tests?

Key Result

Organize pytest tests with fixed data, fixtures, and configs to ensure deterministic, repeatable results.