@pytest.mark.xfail for expected failures - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of a test marked with @pytest.mark.xfail

What will be the test result when running this pytest test marked with @pytest.mark.xfail and the test fails as expected?

PyTest

import pytest

@pytest.mark.xfail
def test_example():
    assert 1 == 2

A The test is reported as failed.

B The test is reported as passed.

C The test is reported as xfailed (expected failure).

D The test is skipped.

💡 Hint

Think about what @pytest.mark.xfail means when the test fails.

When a test marked with @pytest.mark.xfail fails, pytest reports it as xfailed, meaning the failure was expected and does not count as a failure.

❓ assertion

intermediate

Assertion behavior with @pytest.mark.xfail and strict=True

Given this test marked with @pytest.mark.xfail(strict=True), what happens if the test passes unexpectedly?

PyTest

import pytest

@pytest.mark.xfail(strict=True)
def test_strict_xfail():
    assert 1 == 1

A The test is reported as xpassed (unexpected pass).

B The test is reported as passed.

C The test is reported as failed.

D The test is skipped.

💡 Hint

What does strict=True do when the test passes?

With strict=True, if a test marked xfail passes, pytest reports it as xpassed, indicating an unexpected pass.

🔧 Debug

advanced

Identify why a test marked with @pytest.mark.xfail is reported as failed

Why does this test marked with @pytest.mark.xfail fail instead of being reported as xfailed?

PyTest

import pytest

@pytest.mark.xfail(raises=AssertionError)
def test_bug():
    assert 'a' in 'apple'
    raise ValueError('bug')

A Because the test has multiple assertions and only the last fails, pytest reports failure.

B Because the test raises an unexpected exception type (not AssertionError), pytest reports it as failed, not xfailed.

C Because the first assertion passes, pytest ignores the xfail mark.

D Because the test fails, it should be reported as xfailed, so this is a pytest bug.

💡 Hint

Consider what kinds of failures @pytest.mark.xfail expects.

If the test raises an unexpected exception type (not AssertionError), pytest reports it as failed, not xfailed.

🧠 Conceptual

advanced

Purpose of @pytest.mark.xfail in test suites

What is the main purpose of using @pytest.mark.xfail in a test suite?

A To mark tests that are expected to fail due to known bugs or incomplete features.

B To skip tests that are not ready to run yet.

C To mark tests that should always pass regardless of code changes.

D To mark tests that should run slower for performance reasons.

💡 Hint

Think about tests that fail but you want to keep in the suite.

The xfail mark is used to indicate tests that are expected to fail, often due to known bugs or unfinished features, so their failures don't break the test suite.

❓ framework

expert

Behavior of @pytest.mark.xfail with condition and reason

Consider this test marked with @pytest.mark.xfail(condition, reason). What happens if the condition is False?

PyTest

import pytest

@pytest.mark.xfail(condition=False, reason="Known bug")
def test_conditional_xfail():
    assert False

A The test is reported as passed even if it fails.

B The test is skipped because the condition is False.

C The test is reported as xfailed regardless of the condition.

D The test runs normally and failure is reported as a failure, not xfail.

💡 Hint

What does the condition parameter control in xfail?

If the condition is False, the xfail mark is ignored and the test runs normally; failures are reported as failures.

Practice

1. What is the main purpose of using @pytest.mark.xfail in a test?

easy

A. To mark tests that are expected to fail due to known issues

B. To skip tests that are not ready to run

C. To mark tests that should always pass

D. To run tests multiple times automatically

Solution

1. Step 1: Understand the purpose of @pytest.mark.xfail

   This decorator is used to mark tests that are expected to fail because of known bugs or unfinished features.

2. Step 2: Differentiate from other decorators

   Unlike skipping tests, xfail runs the test but expects failure. It does not mark tests to always pass or run multiple times.

3. Final Answer:

   To mark tests that are expected to fail due to known issues -> Option A

4. Quick Check:

   @pytest.mark.xfail = expected failure [OK]

Hint: xfail means test expected to fail, not skip or pass [OK]

Common Mistakes:

• Confusing xfail with skip
• Thinking xfail forces test to pass
• Assuming xfail reruns tests

2. Which of the following is the correct syntax to mark a test as expected to fail with strict checking in pytest?

easy

A. @pytest.mark.skip(strict=True)
def test_example(): pass

B. @pytest.mark.xfail(strict=True)
def test_example(): pass

C. @pytest.xfail(strict=True)
def test_example(): pass

D. @pytest.mark.xfail(strict=False)
def test_example(): pass

Solution

1. Step 1: Recall correct decorator usage

   The correct decorator is @pytest.mark.xfail with parentheses for parameters.

2. Step 2: Understand strict parameter

   Setting strict=True means pytest will fail the test if it unexpectedly passes.

3. Final Answer:

   @pytest.mark.xfail(strict=True)\ndef test_example(): pass -> Option B

4. Quick Check:

   Correct syntax uses @pytest.mark.xfail(strict=True) [OK]

Hint: Use @pytest.mark.xfail(strict=True) with parentheses [OK]

Common Mistakes:

• Using @pytest.xfail instead of @pytest.mark.xfail
• Confusing skip with xfail
• Omitting parentheses for parameters

3. Consider this pytest code snippet:

@pytest.mark.xfail(strict=True)
def test_divide():
    assert 1 / 0 == 0

What will be the test result when running pytest?

medium

A. Test passes because failure is expected

B. Test fails because division by zero raises an error

C. Test is skipped due to xfail

D. Test fails because strict=True causes failure on error

Solution

1. Step 1: Analyze the test code

   The test asserts 1 divided by 0 equals 0, which raises a ZeroDivisionError, causing failure.

2. Step 2: Understand xfail with strict=True

   Since failure is expected, the test is marked as xfail and will pass the test suite. strict=True only fails if the test unexpectedly passes.

3. Final Answer:

   Test passes because failure is expected -> Option A

4. Quick Check:

   xfail with error = pass test [OK]

Hint: xfail passes if test fails; strict=True fails only if test passes [OK]

Common Mistakes:

• Thinking strict=True fails on error
• Confusing xfail with skip
• Expecting test to fail pytest run

4. You wrote this test:

@pytest.mark.xfail(strict=True)
def test_sum():
    assert sum([1, 2, 3]) == 5

When you run pytest, the test fails. What is the most likely reason?

medium

A. The test unexpectedly passed, causing failure due to strict=True

B. The xfail decorator was not applied correctly

C. The test failed because the assertion is incorrect

D. The test was skipped accidentally

Solution

1. Step 1: Check the assertion logic

   The sum of [1, 2, 3] is 6, but the test asserts it equals 5, so the assertion fails.

2. Step 2: Understand strict=True effect

   strict=True causes failure only if the test unexpectedly passes, but here the test fails as expected.

3. Final Answer:

   The test failed because the assertion is incorrect -> Option C

4. Quick Check:

   Wrong assertion causes failure, not strict=True [OK]

Hint: Check assertion values carefully; strict=True fails only on unexpected pass [OK]

Common Mistakes:

• Assuming strict=True causes failure on test fail
• Thinking test was skipped
• Believing decorator syntax is wrong

5. You have a test that sometimes passes and sometimes fails due to a flaky external API. You want pytest to mark it as expected failure but fail the test run if it unexpectedly passes. Which is the best way to write this?

hard

A. @pytest.mark.xfail()
def test_api():
assert call_api() == 'OK'

B. @pytest.mark.skip(reason='Flaky API')
def test_api():
assert call_api() == 'OK'

C. @pytest.mark.xfail(strict=False, reason='Flaky API')
def test_api():
assert call_api() == 'OK'

D. @pytest.mark.xfail(strict=True, reason='Flaky API')
def test_api():
assert call_api() == 'OK'

Solution

1. Step 1: Identify requirement for strict checking

   You want pytest to fail the test run if the test unexpectedly passes, so strict=True is needed.

2. Step 2: Use xfail with reason for clarity

   Marking the test with @pytest.mark.xfail(strict=True, reason='Flaky API') clearly documents the expected failure and enforces strict checking.

3. Final Answer:

   @pytest.mark.xfail(strict=True, reason='Flaky API')\ndef test_api():\n assert call_api() == 'OK' -> Option D

4. Quick Check:

   xfail with strict=True and reason = best practice [OK]

Hint: Use strict=True to catch unexpected passes on flaky tests [OK]

Common Mistakes:

• Using skip instead of xfail
• Omitting strict=True when needed
• Not providing reason for clarity