@pytest.mark.xfail for expected failures - Test Execution Trace

Test Overview

This test uses @pytest.mark.xfail to mark a test that is expected to fail. It verifies that the test fails as expected without causing the whole test suite to fail.

Test Code - PyTest

PyTest

import pytest

@pytest.mark.xfail(reason="Known bug: division by zero")
def test_divide_by_zero():
    result = 10 / 0
    assert result == 0

Execution Trace - 5 Steps

Step	Action	System State	Assertion	Result
1	Test starts	PyTest test runner initializes and loads test_divide_by_zero	—	PASS
2	Test function test_divide_by_zero is called	Inside test_divide_by_zero function	—	PASS
3	Code executes division 10 / 0 causing ZeroDivisionError	Exception ZeroDivisionError raised	—	FAIL
4	PyTest recognizes test is marked xfail and failure is expected	Test marked as expected failure	Test failure matches expected failure condition	PASS
5	Test suite continues without marking overall run as failed	Test suite summary shows xfail for this test	Test suite reports test as xfailed, not failed	PASS

Failure Scenario

Failing Condition: Test marked xfail unexpectedly passes (no error raised)

Execution Trace Quiz - 3 Questions

What does @pytest.mark.xfail do in this test?

A Marks the test as expected to fail without failing the suite

B Skips the test completely

C Makes the test always pass

D Runs the test twice

Key Result

Use @pytest.mark.xfail to mark tests that are known to fail due to bugs or incomplete features. This helps keep the test suite green while tracking expected failures.

Practice

1. What is the main purpose of using @pytest.mark.xfail in a test?

easy

A. To mark tests that are expected to fail due to known issues

B. To skip tests that are not ready to run

C. To mark tests that should always pass

D. To run tests multiple times automatically

Solution

1. Step 1: Understand the purpose of @pytest.mark.xfail

   This decorator is used to mark tests that are expected to fail because of known bugs or unfinished features.

2. Step 2: Differentiate from other decorators

   Unlike skipping tests, xfail runs the test but expects failure. It does not mark tests to always pass or run multiple times.

3. Final Answer:

   To mark tests that are expected to fail due to known issues -> Option A

4. Quick Check:

   @pytest.mark.xfail = expected failure [OK]

Hint: xfail means test expected to fail, not skip or pass [OK]

Common Mistakes:

• Confusing xfail with skip
• Thinking xfail forces test to pass
• Assuming xfail reruns tests

2. Which of the following is the correct syntax to mark a test as expected to fail with strict checking in pytest?

easy

A. @pytest.mark.skip(strict=True)
def test_example(): pass

B. @pytest.mark.xfail(strict=True)
def test_example(): pass

C. @pytest.xfail(strict=True)
def test_example(): pass

D. @pytest.mark.xfail(strict=False)
def test_example(): pass

Solution

1. Step 1: Recall correct decorator usage

   The correct decorator is @pytest.mark.xfail with parentheses for parameters.

2. Step 2: Understand strict parameter

   Setting strict=True means pytest will fail the test if it unexpectedly passes.

3. Final Answer:

   @pytest.mark.xfail(strict=True)\ndef test_example(): pass -> Option B

4. Quick Check:

   Correct syntax uses @pytest.mark.xfail(strict=True) [OK]

Hint: Use @pytest.mark.xfail(strict=True) with parentheses [OK]

Common Mistakes:

• Using @pytest.xfail instead of @pytest.mark.xfail
• Confusing skip with xfail
• Omitting parentheses for parameters

3. Consider this pytest code snippet:

@pytest.mark.xfail(strict=True)
def test_divide():
    assert 1 / 0 == 0

What will be the test result when running pytest?

medium

A. Test passes because failure is expected

B. Test fails because division by zero raises an error

C. Test is skipped due to xfail

D. Test fails because strict=True causes failure on error

Solution

1. Step 1: Analyze the test code

   The test asserts 1 divided by 0 equals 0, which raises a ZeroDivisionError, causing failure.

2. Step 2: Understand xfail with strict=True

   Since failure is expected, the test is marked as xfail and will pass the test suite. strict=True only fails if the test unexpectedly passes.

3. Final Answer:

   Test passes because failure is expected -> Option A

4. Quick Check:

   xfail with error = pass test [OK]

Hint: xfail passes if test fails; strict=True fails only if test passes [OK]

Common Mistakes:

• Thinking strict=True fails on error
• Confusing xfail with skip
• Expecting test to fail pytest run

4. You wrote this test:

@pytest.mark.xfail(strict=True)
def test_sum():
    assert sum([1, 2, 3]) == 5

When you run pytest, the test fails. What is the most likely reason?

medium

A. The test unexpectedly passed, causing failure due to strict=True

B. The xfail decorator was not applied correctly

C. The test failed because the assertion is incorrect

D. The test was skipped accidentally

Solution

1. Step 1: Check the assertion logic

   The sum of [1, 2, 3] is 6, but the test asserts it equals 5, so the assertion fails.

2. Step 2: Understand strict=True effect

   strict=True causes failure only if the test unexpectedly passes, but here the test fails as expected.

3. Final Answer:

   The test failed because the assertion is incorrect -> Option C

4. Quick Check:

   Wrong assertion causes failure, not strict=True [OK]

Hint: Check assertion values carefully; strict=True fails only on unexpected pass [OK]

Common Mistakes:

• Assuming strict=True causes failure on test fail
• Thinking test was skipped
• Believing decorator syntax is wrong

5. You have a test that sometimes passes and sometimes fails due to a flaky external API. You want pytest to mark it as expected failure but fail the test run if it unexpectedly passes. Which is the best way to write this?

hard

A. @pytest.mark.xfail()
def test_api():
assert call_api() == 'OK'

B. @pytest.mark.skip(reason='Flaky API')
def test_api():
assert call_api() == 'OK'

C. @pytest.mark.xfail(strict=False, reason='Flaky API')
def test_api():
assert call_api() == 'OK'

D. @pytest.mark.xfail(strict=True, reason='Flaky API')
def test_api():
assert call_api() == 'OK'

Solution

1. Step 1: Identify requirement for strict checking

   You want pytest to fail the test run if the test unexpectedly passes, so strict=True is needed.

2. Step 2: Use xfail with reason for clarity

   Marking the test with @pytest.mark.xfail(strict=True, reason='Flaky API') clearly documents the expected failure and enforces strict checking.

3. Final Answer:

   @pytest.mark.xfail(strict=True, reason='Flaky API')\ndef test_api():\n assert call_api() == 'OK' -> Option D

4. Quick Check:

   xfail with strict=True and reason = best practice [OK]

Hint: Use strict=True to catch unexpected passes on flaky tests [OK]

Common Mistakes:

• Using skip instead of xfail
• Omitting strict=True when needed
• Not providing reason for clarity