Worker distribution strategies in PyTest

Introduction

Worker distribution strategies help run tests faster by sharing work across multiple helpers called workers.

You have many tests and want them to finish quickly.

You want to use multiple CPU cores to run tests at the same time.

You want to balance test load evenly so no worker is idle.

You want to avoid some workers getting too many slow tests.

You want to speed up testing in continuous integration systems.

Syntax

PyTest

pytest -n <num_workers> --dist=<mode>

-n sets how many workers run tests in parallel.

--dist chooses how tests are shared: loadscope, loadfile, or loadgroup.

Examples

Runs tests with 4 workers, grouping tests by their scope (like test classes or modules).

PyTest

pytest -n 4 --dist=loadscope

Runs tests with 3 workers, distributing tests by whole test files.

PyTest

pytest -n 3 --dist=loadfile

Runs tests with 2 workers, grouping tests by custom groups you define.

PyTest

pytest -n 2 --dist=loadgroup

Sample Program

This simple test runs three checks. Using -n 2 and --dist=loadscope, pytest runs tests in parallel with 2 workers, grouping tests by their scope to balance load.

PyTest

import pytest

@pytest.mark.parametrize('x', [1, 2, 3])
def test_example(x):
    assert x > 0

# Run this command in terminal:
# pytest -n 2 --dist=loadscope

# This runs the tests in parallel using 2 workers, grouping tests by their scope.

Important Notes

Use loadscope to group tests by their class or module, which helps balance load better than just files.

loadfile sends whole test files to workers, which is simpler but may cause imbalance if files differ in size.

loadgroup requires you to define groups, useful for special cases.

Summary

Worker distribution strategies help run tests faster by sharing work across multiple workers.

Use -n to set number of workers and --dist to choose distribution mode.

Common modes are loadscope (group by scope), loadfile (group by file), and loadgroup (custom groups).