Worker distribution strategies in PyTest - Build an Automation Script

Verify pytest worker distribution with multiple tests

Preconditions (2)

Step 1: Run pytest with the -n 2 option to use 2 workers

Step 2: Observe that tests are distributed between the two workers

Step 3: Run pytest with the -n 4 option to use 4 workers

Step 4: Observe that tests are distributed among the four workers

Step 5: Run pytest with the --dist=loadscope option and 2 workers

Step 6: Observe that tests grouped by scope are distributed to workers

✅ Expected Result: Tests are split and run in parallel across the specified number of workers according to the distribution strategy, with no test failures due to worker distribution.

Automation Requirements - pytest

Assertions Needed:

Verify all tests pass when run with multiple workers

Verify tests are executed in parallel by checking timestamps or logs

Verify no test is skipped or duplicated

Best Practices:

Use pytest-xdist plugin for parallel execution

Use fixtures with appropriate scope to avoid conflicts

Avoid shared state between tests to prevent race conditions

Use explicit markers or grouping if using loadscope distribution

Automated Solution

PyTest

import pytest
import time
import threading

execution_log = []

@pytest.fixture(scope='module')
def resource():
    return "shared_resource"

@pytest.mark.parametrize('i', range(4))
def test_parallel_execution(i, resource):
    start = time.time()
    thread_name = threading.current_thread().name
    execution_log.append((i, thread_name, start))
    time.sleep(0.5)  # Simulate work
    end = time.time()
    execution_log.append((i, thread_name, end))
    assert resource == "shared_resource"


def test_all_tests_ran():
    # Check that all 4 tests ran
    test_ids = set(entry[0] for entry in execution_log)
    assert test_ids == {0,1,2,3}

# To run this test suite with parallel workers, use:
# pytest -n 2 --dist=loadscope
# or
# pytest -n 4

# The assertions verify all tests ran and used the shared fixture correctly.

The test suite defines 4 parameterized tests to simulate independent tests.

Each test logs its start and end time with the thread name to show parallel execution.

The shared fixture with module scope ensures no conflicts.

The final test verifies all tests ran by checking the log.

Running with pytest -n 2 or pytest -n 4 uses multiple workers to distribute tests.

The --dist=loadscope option groups tests by scope for distribution.

This setup follows best practices by avoiding shared state conflicts and verifying parallel execution.

Common Mistakes - 4 Pitfalls

Not installing or enabling pytest-xdist plugin

Using shared mutable state without proper isolation

Hardcoding sleep times without synchronization

Not verifying that all tests ran and none were skipped or duplicated

Bonus Challenge

Now add data-driven testing with 3 different inputs for each test and verify parallel execution with workers

Show Hint