@pytest.mark.xfail for expected failures

Introduction

Sometimes tests fail because of known bugs or unfinished features. @pytest.mark.xfail helps us mark these tests so they don't cause confusion.

You know a feature is broken but want to keep the test in your suite.

A bug is reported and you want to track it without failing the whole test run.

You are working on a new feature that is not ready yet but want to prepare tests.

You want to avoid noise from expected failures while focusing on new issues.

Syntax

PyTest

@pytest.mark.xfail(condition=True, reason="", strict=False)
def test_function():
    ...

condition is optional and decides when to mark the test as expected to fail.

reason helps explain why the test is expected to fail.

Examples

This test is always expected to fail.

PyTest

@pytest.mark.xfail
def test_example():
    assert 1 == 2

This test is expected to fail because of a known bug.

PyTest

@pytest.mark.xfail(reason="Bug #123")
def test_bug():
    assert 3 > 5

This test is expected to fail only if the condition is true (here it is false, so test runs normally).

PyTest

@pytest.mark.xfail(condition=(3 > 5), reason="Condition false")
def test_conditional():
    assert 1 == 0

Sample Program

The first test is expected to fail because 1 + 1 is not 3.

The second test is also expected to fail and marked strict, so if it passes, pytest will mark it as an error.

The third test is marked strict but passes, so pytest will report an error because it was expected to fail but didn't.

PyTest

import pytest

@pytest.mark.xfail(reason="Known bug: wrong addition")
def test_addition():
    assert 1 + 1 == 3

@pytest.mark.xfail(strict=True)
def test_strict_fail():
    assert 2 * 2 == 5

@pytest.mark.xfail(strict=True)
def test_strict_pass():
    assert 2 + 2 == 4

Important Notes

Use @pytest.mark.xfail to keep track of tests that fail for known reasons without breaking your test suite.

The strict=True option makes pytest report an error if the test unexpectedly passes.

Always add a reason to explain why the test is expected to fail for better clarity.

Summary

@pytest.mark.xfail marks tests expected to fail.

It helps manage known bugs or unfinished features.

Use strict=True to catch unexpected passes.

Practice

1. What is the main purpose of using @pytest.mark.xfail in a test?

easy

A. To mark tests that are expected to fail due to known issues

B. To skip tests that are not ready to run

C. To mark tests that should always pass

D. To run tests multiple times automatically

Solution

1. Step 1: Understand the purpose of @pytest.mark.xfail

   This decorator is used to mark tests that are expected to fail because of known bugs or unfinished features.

2. Step 2: Differentiate from other decorators

   Unlike skipping tests, xfail runs the test but expects failure. It does not mark tests to always pass or run multiple times.

3. Final Answer:

   To mark tests that are expected to fail due to known issues -> Option A

4. Quick Check:

   @pytest.mark.xfail = expected failure [OK]

Hint: xfail means test expected to fail, not skip or pass [OK]

Common Mistakes:

• Confusing xfail with skip
• Thinking xfail forces test to pass
• Assuming xfail reruns tests

2. Which of the following is the correct syntax to mark a test as expected to fail with strict checking in pytest?

easy

A. @pytest.mark.skip(strict=True)
def test_example(): pass

B. @pytest.mark.xfail(strict=True)
def test_example(): pass

C. @pytest.xfail(strict=True)
def test_example(): pass

D. @pytest.mark.xfail(strict=False)
def test_example(): pass

Solution

1. Step 1: Recall correct decorator usage

   The correct decorator is @pytest.mark.xfail with parentheses for parameters.

2. Step 2: Understand strict parameter

   Setting strict=True means pytest will fail the test if it unexpectedly passes.

3. Final Answer:

   @pytest.mark.xfail(strict=True)\ndef test_example(): pass -> Option B

4. Quick Check:

   Correct syntax uses @pytest.mark.xfail(strict=True) [OK]

Hint: Use @pytest.mark.xfail(strict=True) with parentheses [OK]

Common Mistakes:

• Using @pytest.xfail instead of @pytest.mark.xfail
• Confusing skip with xfail
• Omitting parentheses for parameters

3. Consider this pytest code snippet:

@pytest.mark.xfail(strict=True)
def test_divide():
    assert 1 / 0 == 0

What will be the test result when running pytest?

medium

A. Test passes because failure is expected

B. Test fails because division by zero raises an error

C. Test is skipped due to xfail

D. Test fails because strict=True causes failure on error

Solution

1. Step 1: Analyze the test code

   The test asserts 1 divided by 0 equals 0, which raises a ZeroDivisionError, causing failure.

2. Step 2: Understand xfail with strict=True

   Since failure is expected, the test is marked as xfail and will pass the test suite. strict=True only fails if the test unexpectedly passes.

3. Final Answer:

   Test passes because failure is expected -> Option A

4. Quick Check:

   xfail with error = pass test [OK]

Hint: xfail passes if test fails; strict=True fails only if test passes [OK]

Common Mistakes:

• Thinking strict=True fails on error
• Confusing xfail with skip
• Expecting test to fail pytest run

4. You wrote this test:

@pytest.mark.xfail(strict=True)
def test_sum():
    assert sum([1, 2, 3]) == 5

When you run pytest, the test fails. What is the most likely reason?

medium

A. The test unexpectedly passed, causing failure due to strict=True

B. The xfail decorator was not applied correctly

C. The test failed because the assertion is incorrect

D. The test was skipped accidentally

Solution

1. Step 1: Check the assertion logic

   The sum of [1, 2, 3] is 6, but the test asserts it equals 5, so the assertion fails.

2. Step 2: Understand strict=True effect

   strict=True causes failure only if the test unexpectedly passes, but here the test fails as expected.

3. Final Answer:

   The test failed because the assertion is incorrect -> Option C

4. Quick Check:

   Wrong assertion causes failure, not strict=True [OK]

Hint: Check assertion values carefully; strict=True fails only on unexpected pass [OK]

Common Mistakes:

• Assuming strict=True causes failure on test fail
• Thinking test was skipped
• Believing decorator syntax is wrong

5. You have a test that sometimes passes and sometimes fails due to a flaky external API. You want pytest to mark it as expected failure but fail the test run if it unexpectedly passes. Which is the best way to write this?

hard

A. @pytest.mark.xfail()
def test_api():
assert call_api() == 'OK'

B. @pytest.mark.skip(reason='Flaky API')
def test_api():
assert call_api() == 'OK'

C. @pytest.mark.xfail(strict=False, reason='Flaky API')
def test_api():
assert call_api() == 'OK'

D. @pytest.mark.xfail(strict=True, reason='Flaky API')
def test_api():
assert call_api() == 'OK'

Solution

1. Step 1: Identify requirement for strict checking

   You want pytest to fail the test run if the test unexpectedly passes, so strict=True is needed.

2. Step 2: Use xfail with reason for clarity

   Marking the test with @pytest.mark.xfail(strict=True, reason='Flaky API') clearly documents the expected failure and enforces strict checking.

3. Final Answer:

   @pytest.mark.xfail(strict=True, reason='Flaky API')\ndef test_api():\n assert call_api() == 'OK' -> Option D

4. Quick Check:

   xfail with strict=True and reason = best practice [OK]

Hint: Use strict=True to catch unexpected passes on flaky tests [OK]

Common Mistakes:

• Using skip instead of xfail
• Omitting strict=True when needed
• Not providing reason for clarity