Command-line options in PyTest - Build an Automation Script

Verify pytest command-line options for running tests

Preconditions (2)

Step 1: Open a terminal or command prompt

Step 2: Run 'pytest --maxfail=1 --disable-warnings -q' command in the directory containing test_sample.py

Step 3: Observe the test execution stops after the first failure

Step 4: Observe that warnings are not shown in the output

Step 5: Observe that output is quiet except for test results

✅ Expected Result: pytest runs tests with the specified command-line options: stops after first failure, suppresses warnings, and shows minimal output

Automation Requirements - pytest

Assertions Needed:

Verify test run stops after first failure when --maxfail=1 is used

Verify warnings are suppressed when --disable-warnings is used

Verify output is quiet with -q option

Best Practices:

Use pytest's capsys or caplog fixtures to capture output

Use subprocess module to run pytest commands

Check return codes and output strings for assertions

Avoid hardcoding paths; use pathlib for file paths

Automated Solution

PyTest

import subprocess
import sys
from pathlib import Path
import pytest

def test_pytest_command_line_options(tmp_path):
    # Create a sample test file with two tests, one fails
    test_file = tmp_path / "test_sample.py"
    test_file.write_text(
        """
import pytest

def test_pass():
    assert True

def test_fail():
    assert False
"""
    )

    # Run pytest with --maxfail=1 --disable-warnings -q
    result = subprocess.run(
        [sys.executable, "-m", "pytest", str(test_file), "--maxfail=1", "--disable-warnings", "-q"],
        capture_output=True,
        text=True
    )

    # Check that return code is 1 (because of failure)
    assert result.returncode == 1, f"Expected return code 1 but got {result.returncode}"

    # Check output contains only one failure reported
    # The output should mention '1 failed' and not '2 failed'
    assert "1 failed" in result.stdout or "1 failed" in result.stderr, "Expected exactly one failure"
    assert "2 failed" not in result.stdout and "2 failed" not in result.stderr, "More than one failure reported"

    # Check warnings are not shown
    assert "WARNING" not in result.stdout and "WARNING" not in result.stderr, "Warnings should be suppressed"

    # Check output is quiet (no verbose info)
    # The -q option reduces output, so no 'collected' line
    assert "collected" not in result.stdout, "Output should be quiet without 'collected' line"

This test creates a temporary test file with two tests: one passes and one fails.

It runs pytest as a subprocess with the command-line options --maxfail=1 to stop after the first failure, --disable-warnings to suppress warnings, and -q for quiet output.

The test then asserts the return code is 1, indicating a failure occurred.

It checks the output to confirm only one failure is reported, no warnings appear, and the output is quiet without extra lines like 'collected'.

This approach uses subprocess to simulate real command-line usage and captures output for assertions, following best practices.

Common Mistakes - 4 Pitfalls

Not capturing subprocess output and only checking return code

{'mistake': 'Hardcoding test file paths instead of using temporary directories', 'why_bad': 'Leads to flaky tests and possible file conflicts', 'correct_approach': "Use pytest's tmp_path fixture to create isolated test files"}

{'mistake': 'Running pytest commands without specifying python executable', 'why_bad': 'May run wrong pytest version or fail in some environments', 'correct_approach': "Use sys.executable with '-m pytest' to ensure correct interpreter and pytest module"}

{'mistake': 'Not checking that output is quiet when using -q option', 'why_bad': 'Misses verifying the actual effect of the quiet flag', 'correct_approach': "Assert that verbose lines like 'collected' are not present in output"}

Bonus Challenge

Now add data-driven testing to run the pytest command with different maxfail values: 1, 2, and 3, and verify the behavior accordingly

Show Hint