Test discovery rules in PyTest - Deep Dive

Overview - Test discovery rules

What is it?

Test discovery rules are the guidelines pytest uses to find and run test files and test functions automatically. Pytest looks for files and functions that match specific naming patterns so it knows what to test without manual input. This helps testers run all tests quickly and consistently. Without these rules, you would have to specify every test manually, which is slow and error-prone.

Why it matters

Test discovery rules save time and reduce mistakes by automatically finding tests to run. Without them, testers might miss running some tests or run wrong files, leading to bugs slipping into software. This automation makes testing faster and more reliable, which improves software quality and developer confidence.

Where it fits

Before learning test discovery rules, you should understand basic pytest test writing and running tests manually. After mastering discovery, you can learn about customizing discovery, test selection, and advanced pytest features like fixtures and plugins.

Mental Model

Core Idea

Pytest uses simple naming patterns to automatically find and run your test files and functions without manual listing.

Think of it like...

It's like a librarian who knows to look only for books with titles starting with 'Test' or ending with '_test' on the shelves, so they quickly gather all the books you need without asking you to list each one.

┌───────────────┐
│ Test Discovery│
├───────────────┤
│ File patterns │
│ - test_*.py   │
│ - *_test.py   │
├───────────────┤
│ Function names│
│ - test_*      │
├───────────────┤
│ Pytest finds  │
│ matching files│
│ and functions │
└───────────────┘

Build-Up - 6 Steps

1

Foundation

Basic pytest test naming rules

Concept: Pytest finds test files and functions by their names using simple patterns.

Pytest looks for files named starting with 'test_' or ending with '_test.py'. Inside those files, it looks for functions starting with 'test_'. For example, a file named 'test_math.py' with a function 'test_add()' will be discovered and run automatically.

Result

Pytest runs all functions named 'test_*' inside files matching 'test_*.py' or '*_test.py'.

Understanding these naming rules lets you write tests that pytest will find and run without extra setup.

2

Foundation

How pytest ignores non-test files

Concept: Files and functions not matching the naming patterns are ignored by pytest during discovery.

If a file is named 'math.py' or a function is named 'add_test()', pytest will not run them because they don't match the patterns. This keeps pytest focused only on intended tests.

Result

Only files and functions following the naming rules are included in test runs.

Knowing what pytest ignores helps avoid confusion when tests don't run as expected.

3

Intermediate

Test discovery in subdirectories

🤔Before reading on: do you think pytest finds tests in all folders by default or only the current folder? Commit to your answer.

Concept: Pytest recursively searches subdirectories for test files following the naming rules.

When you run pytest from a folder, it looks inside that folder and all its subfolders for files named 'test_*.py' or '*_test.py'. It then finds test functions inside those files. This means you can organize tests in folders and pytest will still find them.

Result

Tests in nested folders are discovered and run automatically.

Understanding recursive discovery helps you organize tests in folders without losing automatic test running.

4

Intermediate

Customizing test discovery patterns

🤔Before reading on: can you change pytest’s test discovery patterns easily? Commit to yes or no.

Concept: Pytest allows changing the default file and function name patterns using configuration files.

You can create a 'pytest.ini' file and set 'python_files' and 'python_functions' options to tell pytest to look for different names. For example, you could tell pytest to find files named 'check_*.py' instead of 'test_*.py'.

Result

Pytest discovers tests based on your custom naming rules.

Knowing how to customize discovery lets you adapt pytest to different project naming styles.

5

Advanced

How pytest handles test classes and methods

🤔Before reading on: do you think pytest runs any class with test methods or only specific ones? Commit to your answer.

Concept: Pytest discovers test methods inside classes whose names start with 'Test' and that do not have an __init__ method.

Pytest looks for classes named starting with 'Test' and runs their methods starting with 'test_'. Classes with an __init__ method are ignored because pytest expects test classes to be simple containers.

Result

Only test methods inside proper test classes are run.

Understanding this prevents confusion when test methods inside classes are skipped.

6

Expert

Discovery pitfalls with import errors and naming conflicts

🤔Before reading on: do you think pytest will always find tests even if some files have import errors? Commit to yes or no.

Concept: Pytest stops discovering tests in files that raise import errors and can skip tests if naming conflicts or shadowing occur.

If a test file imports a module that fails to load, pytest will error and not run tests in that file. Also, if multiple files or functions have the same name, pytest may run unexpected tests or skip some. Careful naming and fixing import errors are crucial.

Result

Test discovery can silently fail or behave unexpectedly if import errors or naming conflicts exist.

Knowing these pitfalls helps debug why some tests are missing or errors appear during discovery.

Under the Hood

Pytest uses Python's built-in 'os' and 'importlib' modules to scan directories and import test files dynamically. It applies pattern matching on filenames and function names to decide what to load. When a test run starts, pytest walks the directory tree, matches files, imports them as modules, then inspects their contents for test functions and classes. It builds an internal list of test items to run based on these rules.

Why designed this way?

Pytest was designed for simplicity and flexibility. Using naming patterns avoids complex configuration and lets users write tests naturally. Dynamic importing allows pytest to run tests without manual registration. This design balances ease of use with power, avoiding the need for explicit test lists or decorators.

Start
  │
  ▼
Scan directory tree
  │
  ▼
Match files by name (test_*.py, *_test.py)
  │
  ▼
Import matched files
  │
  ▼
Inspect module contents
  │
  ▼
Select functions starting with 'test_'
  │
  ▼
Select classes starting with 'Test' without __init__
  │
  ▼
Select methods starting with 'test_'
  │
  ▼
Build test list
  │
  ▼
Run tests

Myth Busters - 4 Common Misconceptions

Quick: Does pytest run functions named 'check_something' by default? Commit to yes or no.

Common Belief: Pytest runs any function that looks like a test, even if it doesn't start with 'test_'.

Reality: Pytest only runs functions starting with 'test_' by default. Functions named differently are ignored unless configured.

Why it matters: If you name test functions incorrectly, pytest will skip them silently, causing missing test coverage.

Quick: Will pytest find tests in files named 'mytests.py'? Commit to yes or no.

Common Belief: Pytest finds tests in any Python file regardless of its name.

Reality: Pytest only discovers tests in files matching 'test_*.py' or '*_test.py' by default. Files named differently are ignored.

Why it matters: Tests in files with wrong names won't run, leading to false confidence in test completeness.

Quick: If a test file has an import error, will pytest run other tests in that file? Commit to yes or no.

Common Belief: Pytest runs all tests it can, even if some imports fail.

Reality: Pytest stops loading a test file if it encounters import errors, so no tests in that file run.

Why it matters: Import errors can silently block tests, causing missing test runs and hidden bugs.

Quick: Does pytest run test methods in any class? Commit to yes or no.

Common Belief: Pytest runs test methods in all classes regardless of class name or structure.

Reality: Pytest only runs test methods in classes named starting with 'Test' and without an __init__ method.

Why it matters: Test methods in other classes are ignored, which can confuse testers expecting them to run.

Expert Zone

1

Pytest’s discovery can be influenced by plugins that add or change patterns, which can cause unexpected test runs if not understood.

2

Test discovery order is not guaranteed; relying on test execution order can cause flaky tests.

3

Using __init__.py files in test directories can affect discovery by changing package structure and import behavior.

When NOT to use

If your project requires running tests with non-standard naming or complex selection logic, consider using pytest’s explicit test collection hooks or other test runners like unittest or nose that allow manual test registration.

Production Patterns

In large projects, teams often customize pytest.ini to match their naming conventions and use markers to select tests. They organize tests in folders by feature and rely on recursive discovery. Continuous integration systems run pytest with options to collect and report tests efficiently.

Connections

Continuous Integration (CI)

Builds-on

Understanding pytest test discovery helps configure CI pipelines to run all tests automatically, ensuring code quality on every change.

Static Code Analysis

Complementary

While static analysis checks code without running it, test discovery ensures dynamic tests run correctly; combining both improves software reliability.

File System Indexing

Similar pattern

Test discovery’s pattern matching and recursive search is like how file system indexers find files by name and type, showing how software uses common search strategies.

Common Pitfalls

#1 Tests not running because function names don't start with 'test_'.

Wrong approach:

def check_addition():
assert 1 + 1 == 2

Correct approach:

def test_addition():
assert 1 + 1 == 2

Root cause:

Misunderstanding pytest’s naming rules for test functions.

#2 Test files named incorrectly so pytest ignores them.

Wrong approach:

# File named math_tests.py

def test_sum():
assert sum([1,2]) == 3

Correct approach:

# File renamed to test_math.py

def test_sum():
assert sum([1,2]) == 3

Root cause:

Not following pytest’s file naming conventions.

#3 Tests skipped due to import errors in test files.

Wrong approach:

# test_example.py
import missing_module

def test_func():
assert True

Correct approach:

# test_example.py
# Ensure all imports exist
import math

def test_func():
assert True

Root cause:

Ignoring import errors that block test discovery.

Key Takeaways

Pytest finds tests automatically by looking for files and functions with specific name patterns.

Only files named 'test_*.py' or '*_test.py' and functions starting with 'test_' are discovered by default.

Pytest searches recursively in subdirectories, allowing organized test folders.

Import errors or wrong naming can silently prevent tests from running, so careful naming and error checking are essential.

Advanced users can customize discovery patterns and understand internal mechanics to tailor pytest to complex projects.