Command-line options in PyTest - Build an Automation Script

Verify pytest command-line options for running tests

Preconditions (2)

Step 1: Open a terminal or command prompt

Step 2: Run 'pytest --maxfail=1 --disable-warnings -q' command in the directory containing test_sample.py

Step 3: Observe the test execution stops after the first failure

Step 4: Observe that warnings are not shown in the output

Step 5: Observe that output is quiet except for test results

✅ Expected Result: pytest runs tests with the specified command-line options: stops after first failure, suppresses warnings, and shows minimal output

Automation Requirements - pytest

Assertions Needed:

Verify test run stops after first failure when --maxfail=1 is used

Verify warnings are suppressed when --disable-warnings is used

Verify output is quiet with -q option

Best Practices:

Use pytest's capsys or caplog fixtures to capture output

Use subprocess module to run pytest commands

Check return codes and output strings for assertions

Avoid hardcoding paths; use pathlib for file paths

Automated Solution

PyTest

import subprocess
import sys
from pathlib import Path
import pytest

def test_pytest_command_line_options(tmp_path):
    # Create a sample test file with two tests, one fails
    test_file = tmp_path / "test_sample.py"
    test_file.write_text(
        """
import pytest

def test_pass():
    assert True

def test_fail():
    assert False
"""
    )

    # Run pytest with --maxfail=1 --disable-warnings -q
    result = subprocess.run(
        [sys.executable, "-m", "pytest", str(test_file), "--maxfail=1", "--disable-warnings", "-q"],
        capture_output=True,
        text=True
    )

    # Check that return code is 1 (because of failure)
    assert result.returncode == 1, f"Expected return code 1 but got {result.returncode}"

    # Check output contains only one failure reported
    # The output should mention '1 failed' and not '2 failed'
    assert "1 failed" in result.stdout or "1 failed" in result.stderr, "Expected exactly one failure"
    assert "2 failed" not in result.stdout and "2 failed" not in result.stderr, "More than one failure reported"

    # Check warnings are not shown
    assert "WARNING" not in result.stdout and "WARNING" not in result.stderr, "Warnings should be suppressed"

    # Check output is quiet (no verbose info)
    # The -q option reduces output, so no 'collected' line
    assert "collected" not in result.stdout, "Output should be quiet without 'collected' line"

This test creates a temporary test file with two tests: one passes and one fails.

It runs pytest as a subprocess with the command-line options --maxfail=1 to stop after the first failure, --disable-warnings to suppress warnings, and -q for quiet output.

The test then asserts the return code is 1, indicating a failure occurred.

It checks the output to confirm only one failure is reported, no warnings appear, and the output is quiet without extra lines like 'collected'.

This approach uses subprocess to simulate real command-line usage and captures output for assertions, following best practices.

Common Mistakes - 4 Pitfalls

Not capturing subprocess output and only checking return code

{'mistake': 'Hardcoding test file paths instead of using temporary directories', 'why_bad': 'Leads to flaky tests and possible file conflicts', 'correct_approach': "Use pytest's tmp_path fixture to create isolated test files"}

{'mistake': 'Running pytest commands without specifying python executable', 'why_bad': 'May run wrong pytest version or fail in some environments', 'correct_approach': "Use sys.executable with '-m pytest' to ensure correct interpreter and pytest module"}

{'mistake': 'Not checking that output is quiet when using -q option', 'why_bad': 'Misses verifying the actual effect of the quiet flag', 'correct_approach': "Assert that verbose lines like 'collected' are not present in output"}

Bonus Challenge

Now add data-driven testing to run the pytest command with different maxfail values: 1, 2, and 3, and verify the behavior accordingly

Show Hint

Practice

1. What does the pytest command-line option -v do when running tests?

easy

A. Runs tests only once without reruns

B. Runs tests in quiet mode, suppressing output

C. Generates a test coverage report

D. Runs tests in verbose mode, showing each test name

Solution

1. Step 1: Understand the -v option in pytest

   The -v option stands for verbose mode, which means pytest will show detailed information about each test it runs.

2. Step 2: Compare with other options

   Options like quiet mode or coverage report are different flags, so -v specifically increases output detail.

3. Final Answer:

   Runs tests in verbose mode, showing each test name -> Option D

4. Quick Check:

   Verbose = detailed output [OK]

Hint: Verbose means more details shown during test run [OK]

Common Mistakes:

• Confusing verbose with quiet mode
• Thinking it runs tests only once
• Mixing it with coverage report option

2. Which of the following is the correct syntax to run pytest and stop after the first failure?

easy

A. pytest -x

B. pytest --stop-on-failure

C. pytest --failfast=false

D. pytest -f

Solution

1. Step 1: Recall pytest option to stop on first failure

   The correct option is -x, which tells pytest to stop after the first test failure.

2. Step 2: Check other options for correctness

   --stop-on-failure and --failfast=false are invalid, and -f is not a recognized pytest option.

3. Final Answer:

   pytest -x -> Option A

4. Quick Check:

   Stop on first failure = -x [OK]

Hint: Use -x to stop pytest after first failure [OK]

Common Mistakes:

• Using long option names that don't exist
• Confusing -f with -x
• Assuming --failfast=false stops tests

3. Given the command pytest -k 'test_add or test_subtract' -v, what will pytest do?

medium

A. Run all tests except those named test_add or test_subtract

B. Run only tests named test_add or test_subtract with verbose output

C. Run tests with names containing 'test' only

D. Run tests and stop after first failure

Solution

1. Step 1: Understand the -k option

   The -k option filters tests by substring expressions. Here, it selects tests named test_add or test_subtract.

2. Step 2: Understand the -v option

   The -v option runs tests in verbose mode, showing detailed test names and results.

3. Final Answer:

   Run only tests named test_add or test_subtract with verbose output -> Option B

4. Quick Check:

   -k filters tests, -v shows details [OK]

Hint: -k filters tests by name, -v shows details [OK]

Common Mistakes:

• Thinking -k excludes tests instead of including
• Ignoring the verbose flag
• Assuming it stops on failure

4. You run pytest --maxfail=2 -v but pytest does not stop after 2 failures. What is the likely cause?

medium

A. You need to add -x to stop after max failures

B. The option --maxfail requires a space before the number

C. The --maxfail option is deprecated and ignored

D. You must use --maxfail=2 without -v

Solution

1. Step 1: Understand --maxfail behavior

   The --maxfail option sets how many failures to allow before stopping, but it requires -x to actually stop the test run.

2. Step 2: Check command usage

   Without -x, pytest will report max failures but continue running all tests.

3. Final Answer:

   You need to add -x to stop after max failures -> Option A

4. Quick Check:

   --maxfail needs -x to stop tests [OK]

Hint: Add -x with --maxfail to stop tests early [OK]

Common Mistakes:

• Thinking --maxfail alone stops tests
• Adding space between --maxfail and number
• Assuming -v disables maxfail

5. You want to run pytest only on tests marked as slow and generate a JUnit XML report saved as results.xml. Which command is correct?

hard

A. pytest --only-mark slow --junitxml results.xml

B. pytest --mark slow --xml=results.xml

C. pytest -m slow --junitxml=results.xml

D. pytest -m slow -x --junit=results.xml

Solution

1. Step 1: Use -m to select tests by marker

   The -m option filters tests by marker name, so -m slow runs only tests marked as slow.

2. Step 2: Use --junitxml=filename to generate report

   The --junitxml=results.xml option saves test results in JUnit XML format to the specified file.

3. Step 3: Verify other options are invalid

   Options like --mark, --xml, and --only-mark are not valid pytest flags.

4. Final Answer:

   pytest -m slow --junitxml=results.xml -> Option C

5. Quick Check:

   -m for marker, --junitxml for report [OK]

Hint: Use -m for markers and --junitxml for reports [OK]

Common Mistakes:

• Using incorrect flags like --mark or --xml
• Forgetting = between --junitxml and filename
• Adding -x which stops tests unnecessarily