Running tests by marker (-m) in PyTest - Framework Patterns

Framework Mode - Running tests by marker (-m)

Folder Structure

project-root/
├── tests/
│   ├── test_login.py
│   ├── test_signup.py
│   ├── test_profile.py
│   └── __init__.py
├── pytest.ini
├── conftest.py
└── utils/
    └── helpers.py
  

Test Framework Layers

• Tests Layer: Contains test files with test functions or classes. Markers are applied here to categorize tests (e.g., @pytest.mark.smoke).
• Fixtures & Config Layer: conftest.py holds fixtures and hooks shared across tests.
• Utilities Layer: Helper functions and reusable code in utils/helpers.py.
• Configuration Layer: pytest.ini defines markers and test run options.

Configuration Patterns

Use pytest.ini to declare custom markers to avoid warnings and organize tests:

[pytest]
markers =
    smoke: quick smoke tests
    regression: full regression tests
    slow: tests that take longer time

Run tests by marker using the -m option:

pytest -m smoke

This runs only tests marked with @pytest.mark.smoke.

Combine markers with and, or, and not:

pytest -m "smoke and not slow"

Use environment variables or command line options to select markers dynamically if needed.

Test Reporting and CI/CD Integration

• Use pytest's built-in reporting to see which tests ran and their results.
• Integrate with CI tools (GitHub Actions, Jenkins) to run marker-based tests selectively, e.g., smoke tests on pull requests.
• Generate reports (JUnit XML, HTML) to visualize test outcomes filtered by markers.
• Example GitHub Action step to run smoke tests:

  pytest -m smoke --junitxml=reports/smoke-results.xml

Best Practices

• Always declare custom markers in pytest.ini to keep the framework clean and avoid warnings.
• Use markers to group tests by purpose (smoke, regression, slow) for flexible test runs.
• Keep marker names simple and meaningful for easy understanding by all team members.
• Combine markers logically with and, or, and not to fine-tune test selection.
• Integrate marker-based runs into CI pipelines to optimize test execution time and feedback speed.

Self Check

Where in this folder structure would you add a new test file for API tests that you want to mark as regression?

Key Result

Use pytest markers declared in pytest.ini to categorize and run tests selectively with the -m option.

Practice

1. What does the -m option do when running pytest?

easy

A. Runs only tests with the specified marker

B. Runs tests in a specific directory

C. Runs tests in parallel

D. Generates a test report

Solution

1. Step 1: Understand pytest markers

   Markers are labels you add to tests to group them logically.

2. Step 2: Use -m option

   The -m option runs only tests that have the specified marker label.

3. Final Answer:

   Runs only tests with the specified marker -> Option A

4. Quick Check:

   pytest -m runs tests by marker = A [OK]

Hint: Remember: -m filters tests by marker name [OK]

Common Mistakes:

• Confusing -m with directory or file options
• Thinking -m runs all tests regardless of marker
• Assuming -m runs tests in parallel

2. Which of the following is the correct syntax to run pytest tests marked as slow?

easy

A. pytest --marker slow

B. pytest -m slow

C. pytest -marker slow

D. pytest --m slow

Solution

1. Step 1: Recall pytest command options

   The correct option to select tests by marker is -m, not --marker or others.

2. Step 2: Match syntax to run tests with marker

   The correct command is pytest -m slow to run tests marked as slow.

3. Final Answer:

   pytest -m slow -> Option B

4. Quick Check:

   Use single dash -m for marker filter = D [OK]

Hint: Use single dash -m, not --marker [OK]

Common Mistakes:

• Using --marker instead of -m
• Adding extra dashes like --m
• Confusing option names

3. Given these tests marked as fast and db:

@pytest.mark.fast
def test_a(): pass

@pytest.mark.db
def test_b(): pass

@pytest.mark.fast
def test_c(): pass

What tests run with the command pytest -m "fast and not db"?

medium

A. test_b only

B. No tests run

C. test_a, test_b, and test_c

D. test_a and test_c

Solution

1. Step 1: Understand the marker expression

   The command runs tests marked with fast AND NOT marked with db.

2. Step 2: Check each test's markers

   test_a and test_c have fast only; test_b has db only.

3. Final Answer:

   test_a and test_c -> Option D

4. Quick Check:

   fast and not db = test_a, test_c [OK]

Hint: Use logical operators to combine markers correctly [OK]

Common Mistakes:

• Including tests marked only with db
• Ignoring the 'not' operator
• Assuming all tests run

4. You run pytest -m slow but no tests run, even though some tests have @pytest.mark.slow. What is the most likely cause?

medium

A. Markers are not registered in pytest.ini

B. You used -m instead of --m

C. The tests are skipped by default

D. The test files are missing

Solution

1. Step 1: Check marker registration

   Pytest requires custom markers to be registered in pytest.ini to avoid warnings and ensure correct filtering.

2. Step 2: Understand effect of missing registration

   If markers are not registered, pytest may ignore them or not run tests filtered by them.

3. Final Answer:

   Markers are not registered in pytest.ini -> Option A

4. Quick Check:

   Unregistered markers cause no tests run = C [OK]

Hint: Always register custom markers in pytest.ini [OK]

Common Mistakes:

• Confusing -m and --m options
• Assuming tests are skipped by default
• Not checking test file presence

5. You have tests marked as api, ui, and db. You want to run tests that are either api or ui, but not db. Which command achieves this?

hard

A. pytest -m "api or (ui and not db)"

B. pytest -m "api or ui and not db"

C. pytest -m "(api or ui) and not db"

D. pytest -m "api and ui and not db"

Solution

1. Step 1: Understand operator precedence

   Logical AND has higher precedence than OR, so parentheses are needed to group api or ui together.

2. Step 2: Analyze each option

   pytest -m "(api or ui) and not db" correctly groups (api or ui) and excludes db with and not db.

3. Final Answer:

   pytest -m "(api or ui) and not db" -> Option C

4. Quick Check:

   Use parentheses to group OR conditions = A [OK]

Hint: Use parentheses to group markers with OR before AND [OK]

Common Mistakes:

• Ignoring operator precedence without parentheses
• Using AND instead of OR between api and ui
• Including db tests by mistake