Command-line options in PyTest - Deep Dive

Overview - Command-line options

What is it?

Command-line options in pytest are special flags or arguments you add when running tests from the terminal. They change how pytest behaves, like selecting which tests to run, showing more details, or stopping early. These options help customize test runs without changing the test code itself.

Why it matters

Without command-line options, running tests would be rigid and slow. You couldn't easily pick specific tests or get detailed info when something fails. This would waste time and make debugging harder. Command-line options let you work smarter by controlling test runs to fit your needs.

Where it fits

Before learning command-line options, you should know how to write basic pytest tests and run them simply. After mastering options, you can explore pytest plugins and advanced configuration files to automate and extend testing.

Mental Model

Core Idea

Command-line options are like remote controls that let you change how pytest runs tests without touching the test code.

Think of it like...

Imagine watching a movie with a remote control. You can pause, rewind, or skip scenes without changing the movie itself. Command-line options are like that remote for your tests.

┌─────────────────────────────┐
│        pytest command        │
├─────────────┬───────────────┤
│ Base command│ pytest         │
│ Options     │ -k 'test_name' │
│             │ --maxfail=2    │
│             │ -v             │
└─────────────┴───────────────┘

Options modify test selection, output, and behavior.

Build-Up - 7 Steps

1

Foundation

Running pytest with no options

Concept: Learn how pytest runs tests by default without any extra options.

Open your terminal and run `pytest` in a folder with test files. Pytest will find all tests and run them, showing a simple summary of passed or failed tests.

Result

Pytest runs all tests it finds and shows a dot for each passed test or an F for failures.

Understanding the default behavior sets the stage for why and how command-line options can improve your testing experience.

2

Foundation

Basic option: selecting tests by name

Concept: Use the `-k` option to run only tests matching a keyword in their names.

Run `pytest -k 'keyword'` to run tests whose names contain 'keyword'. For example, `pytest -k 'login'` runs tests with 'login' in their names.

Result

Only tests matching the keyword run, saving time and focusing on relevant tests.

Knowing how to pick specific tests quickly helps when debugging or developing features.

3

Intermediate

Controlling output verbosity

🤔Before reading on: do you think adding -v will show more or less test details? Commit to your answer.

Concept: The `-v` (verbose) option makes pytest show detailed info about each test run.

Run `pytest -v` to see each test's full name and its pass/fail status instead of just dots or letters.

Result

Test output lists each test individually, making it easier to track progress and identify which tests ran.

Understanding verbosity helps you get the right amount of info for your current task, balancing detail and noise.

4

Intermediate

Stopping after failures

🤔Before reading on: if you set --maxfail=1, will pytest stop after one failure or keep running all tests? Commit to your answer.

Concept: The `--maxfail` option stops pytest after a set number of test failures.

Run `pytest --maxfail=1` to stop testing as soon as one test fails. This saves time when you want to fix the first problem before continuing.

Result

Pytest stops running tests after the first failure, showing only that failure in detail.

Knowing how to stop early prevents wasting time on tests that might fail for the same root cause.

5

Intermediate

Running tests with markers

🤔Before reading on: do you think markers are used to group tests or to change test code? Commit to your answer.

Concept: Markers label tests so you can select groups of tests using the `-m` option.

Add `@pytest.mark.slow` above slow tests. Run `pytest -m slow` to run only those tests. This helps organize tests by categories like speed or feature.

Result

Only tests with the chosen marker run, making it easy to run subsets of tests.

Markers combined with command-line options give powerful control over test selection without changing test logic.

6

Advanced

Customizing output with --tb option

🤔Before reading on: do you think --tb=short shows more or less error detail than the default? Commit to your answer.

Concept: The `--tb` option controls how much traceback detail pytest shows on failures.

Run `pytest --tb=short` for a brief error traceback or `--tb=long` for full details. This helps focus on relevant info when debugging.

Result

Test failure output changes in length and detail, helping you read errors faster or deeper.

Adjusting traceback detail helps balance speed and depth when investigating test failures.

7

Expert

Combining options and plugins for efficiency

🤔Before reading on: can combining options and plugins create conflicts or only improve testing? Commit to your answer.

Concept: You can combine multiple command-line options and pytest plugins to tailor test runs precisely and automate workflows.

Example: `pytest -v -k 'login' --maxfail=2 --tb=short` runs verbose login tests, stops after two failures, and shows short tracebacks. Plugins like pytest-xdist can run tests in parallel with options.

Result

Test runs become faster, more focused, and easier to debug, improving developer productivity.

Mastering option combinations and plugins unlocks pytest's full power for real-world testing challenges.

Under the Hood

When you run pytest with command-line options, the pytest runner parses these options first. It adjusts internal settings like which tests to collect, how to display output, and when to stop running tests. Options are processed by pytest's internal parser and passed to plugins or core code that control test discovery, execution, and reporting.

Why designed this way?

Pytest was designed to be flexible and user-friendly. Command-line options let users customize behavior without changing test code, supporting many workflows. This separation keeps tests clean and lets users adapt pytest to different projects and needs easily.

┌───────────────┐
│ User runs     │
│ pytest + opts │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Option parser │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Test collector│
│ & selector   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Test executor │
│ (runs tests)  │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Reporter      │
│ (shows output)│
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does using -k 'test' run tests only with exact name 'test'? Commit to yes or no.

Common Belief: The -k option runs tests only if their name exactly matches the keyword.

Reality: The -k option runs tests whose names contain the keyword anywhere, not just exact matches.

Why it matters: Misunderstanding this can cause running too many or too few tests, wasting time or missing bugs.

Quick: Does --maxfail=0 mean no limit on failures or stop immediately? Commit to your answer.

Common Belief: Setting --maxfail=0 disables stopping and runs all tests regardless of failures.

Reality: Setting --maxfail=0 actually means stop immediately on the first failure.

Why it matters: Wrong assumptions here can cause tests to stop too early or run too long, affecting debugging speed.

Quick: Does -v make tests run slower or just change output? Commit to your answer.

Common Belief: The -v option slows down tests because it adds extra processing.

Reality: The -v option only changes output verbosity; it does not affect test execution speed significantly.

Why it matters: Believing this may discourage using helpful verbose output, making debugging harder.

Quick: Can markers run tests that are not marked if you use -m? Commit to yes or no.

Common Belief: Using -m runs all tests, ignoring markers if no marker matches.

Reality: Using -m runs only tests with the specified marker; unmarked tests are skipped.

Why it matters: Misusing markers can cause important tests to be skipped unintentionally.

Expert Zone

1

Some command-line options can conflict or override each other, so order and combination matter in complex test runs.

2

Plugins can add their own command-line options that integrate seamlessly, but may change default behaviors unexpectedly.

3

Using environment variables alongside command-line options can create subtle differences in test runs that are hard to debug.

When NOT to use

Command-line options are less suitable for permanent test configurations; for that, use pytest.ini or other config files. Also, avoid overusing options in CI pipelines where consistent runs are critical; prefer fixed configs.

Production Patterns

In real projects, teams use command-line options in scripts and CI pipelines to run subsets of tests, control output for logs, and stop early on failures. Combining options with plugins like pytest-xdist for parallel runs is common to speed up testing.

Connections

Unix shell scripting

Command-line options in pytest follow the same pattern as Unix command flags that modify program behavior.

Understanding Unix command-line flags helps grasp pytest options quickly since both use similar syntax and concepts.

Feature toggles in software development

Both command-line options and feature toggles control behavior without changing core code.

Knowing how feature toggles work clarifies why command-line options are powerful for flexible testing without code changes.

Remote control devices

Command-line options act like remote controls for software, changing behavior on demand.

Seeing command-line options as remote controls helps appreciate their role in customizing software behavior dynamically.

Common Pitfalls

#1 Running all tests when you only want a subset.

Wrong approach:

pytest -k 'login_test' # but actual test names are 'test_login' so no tests run

Correct approach:

pytest -k 'login' # matches any test with 'login' in the name

Root cause:

Misunderstanding how the -k option matches substrings, expecting exact matches.

#2 Expecting --maxfail=0 to run all tests without stopping.

Wrong approach:

pytest --maxfail=0 # stops immediately on first failure

Correct approach:

pytest --maxfail=9999 # effectively no limit on failures

Root cause:

Confusing the meaning of zero in --maxfail; zero means stop immediately.

#3 Using -v and expecting it to slow down tests significantly.

Wrong approach:

pytest -v # thinking this causes slow tests

Correct approach:

pytest -v # only changes output verbosity, no major speed impact

Root cause:

Assuming output detail affects test execution speed.

Key Takeaways

Command-line options let you control pytest behavior dynamically without changing test code.

Options like -k, -m, -v, and --maxfail help select tests, adjust output, and manage test runs efficiently.

Understanding how options work together prevents wasted time and missed bugs during testing.

Advanced use combines options with plugins and config files for powerful, flexible test automation.

Misunderstanding option behavior can cause skipped tests or premature stops, so careful use is essential.

Practice

1. What does the pytest command-line option -v do when running tests?

easy

A. Runs tests only once without reruns

B. Runs tests in quiet mode, suppressing output

C. Generates a test coverage report

D. Runs tests in verbose mode, showing each test name

Solution

1. Step 1: Understand the -v option in pytest

   The -v option stands for verbose mode, which means pytest will show detailed information about each test it runs.

2. Step 2: Compare with other options

   Options like quiet mode or coverage report are different flags, so -v specifically increases output detail.

3. Final Answer:

   Runs tests in verbose mode, showing each test name -> Option D

4. Quick Check:

   Verbose = detailed output [OK]

Hint: Verbose means more details shown during test run [OK]

Common Mistakes:

• Confusing verbose with quiet mode
• Thinking it runs tests only once
• Mixing it with coverage report option

2. Which of the following is the correct syntax to run pytest and stop after the first failure?

easy

A. pytest -x

B. pytest --stop-on-failure

C. pytest --failfast=false

D. pytest -f

Solution

1. Step 1: Recall pytest option to stop on first failure

   The correct option is -x, which tells pytest to stop after the first test failure.

2. Step 2: Check other options for correctness

   --stop-on-failure and --failfast=false are invalid, and -f is not a recognized pytest option.

3. Final Answer:

   pytest -x -> Option A

4. Quick Check:

   Stop on first failure = -x [OK]

Hint: Use -x to stop pytest after first failure [OK]

Common Mistakes:

• Using long option names that don't exist
• Confusing -f with -x
• Assuming --failfast=false stops tests

3. Given the command pytest -k 'test_add or test_subtract' -v, what will pytest do?

medium

A. Run all tests except those named test_add or test_subtract

B. Run only tests named test_add or test_subtract with verbose output

C. Run tests with names containing 'test' only

D. Run tests and stop after first failure

Solution

1. Step 1: Understand the -k option

   The -k option filters tests by substring expressions. Here, it selects tests named test_add or test_subtract.

2. Step 2: Understand the -v option

   The -v option runs tests in verbose mode, showing detailed test names and results.

3. Final Answer:

   Run only tests named test_add or test_subtract with verbose output -> Option B

4. Quick Check:

   -k filters tests, -v shows details [OK]

Hint: -k filters tests by name, -v shows details [OK]

Common Mistakes:

• Thinking -k excludes tests instead of including
• Ignoring the verbose flag
• Assuming it stops on failure

4. You run pytest --maxfail=2 -v but pytest does not stop after 2 failures. What is the likely cause?

medium

A. You need to add -x to stop after max failures

B. The option --maxfail requires a space before the number

C. The --maxfail option is deprecated and ignored

D. You must use --maxfail=2 without -v

Solution

1. Step 1: Understand --maxfail behavior

   The --maxfail option sets how many failures to allow before stopping, but it requires -x to actually stop the test run.

2. Step 2: Check command usage

   Without -x, pytest will report max failures but continue running all tests.

3. Final Answer:

   You need to add -x to stop after max failures -> Option A

4. Quick Check:

   --maxfail needs -x to stop tests [OK]

Hint: Add -x with --maxfail to stop tests early [OK]

Common Mistakes:

• Thinking --maxfail alone stops tests
• Adding space between --maxfail and number
• Assuming -v disables maxfail

5. You want to run pytest only on tests marked as slow and generate a JUnit XML report saved as results.xml. Which command is correct?

hard

A. pytest --only-mark slow --junitxml results.xml

B. pytest --mark slow --xml=results.xml

C. pytest -m slow --junitxml=results.xml

D. pytest -m slow -x --junit=results.xml

Solution

1. Step 1: Use -m to select tests by marker

   The -m option filters tests by marker name, so -m slow runs only tests marked as slow.

2. Step 2: Use --junitxml=filename to generate report

   The --junitxml=results.xml option saves test results in JUnit XML format to the specified file.

3. Step 3: Verify other options are invalid

   Options like --mark, --xml, and --only-mark are not valid pytest flags.

4. Final Answer:

   pytest -m slow --junitxml=results.xml -> Option C

5. Quick Check:

   -m for marker, --junitxml for report [OK]

Hint: Use -m for markers and --junitxml for reports [OK]

Common Mistakes:

• Using incorrect flags like --mark or --xml
• Forgetting = between --junitxml and filename
• Adding -x which stops tests unnecessarily