Coverage thresholds in PyTest - Build an Automation Script

Verify code coverage meets minimum threshold using pytest

Preconditions (2)

Step 1: Run pytest with coverage enabled using the command: pytest --cov=your_package_name

Step 2: Check the coverage report summary in the console output

Step 3: Verify that the total coverage percentage is at least 80%

✅ Expected Result: The coverage report shows total coverage percentage of 80% or higher, indicating tests cover at least 80% of the code.

Automation Requirements - pytest

Assertions Needed:

Assert that coverage percentage is >= 80%

Best Practices:

Use pytest-cov plugin for coverage measurement

Use pytest hooks or coverage API to programmatically check coverage

Fail the test if coverage threshold is not met

Automated Solution

PyTest

import pytest
from coverage import Coverage

# This test runs coverage programmatically and asserts threshold

def test_coverage_threshold():
    cov = Coverage(source=["your_package_name"])
    cov.start()

    # Import the module to run its code coverage
    import your_package_name

    # Run tests programmatically using pytest main
    result = pytest.main(["tests/"])

    cov.stop()
    cov.save()

    total_coverage = cov.report(show_missing=False)

    assert total_coverage >= 80, f"Coverage {total_coverage}% is below threshold of 80%"

This test uses the coverage module to start coverage measurement on the target package. It then runs all tests in the tests/ directory using pytest.main(). After tests finish, it stops coverage and reports the total coverage percentage. The assertion checks if coverage is at least 80%. If not, the test fails with a clear message.

This approach automates coverage threshold verification inside a test, so coverage enforcement is part of the test suite.

Common Mistakes - 3 Pitfalls

Running pytest with coverage from command line but not asserting coverage in code

Hardcoding coverage threshold in multiple places

Not stopping or saving coverage before reporting

Bonus Challenge

Now add data-driven testing to verify coverage thresholds for multiple packages with different minimum percentages

Show Hint

Practice

1. What does setting a coverage threshold in pytest ensure?

easy

A. Only tests with coverage above 90% are executed

B. Tests run faster by skipping some checks

C. The test run fails if coverage is below a set percentage

D. Coverage reports are hidden from the user

Solution

1. Step 1: Understand coverage threshold purpose

   Coverage thresholds set a minimum coverage percentage to maintain test quality.

2. Step 2: Effect of coverage below threshold

   If coverage is below the threshold, pytest fails the test run to alert missing tests.

3. Final Answer:

   The test run fails if coverage is below a set percentage -> Option C

4. Quick Check:

   Coverage threshold = fail if below limit [OK]

Hint: Coverage threshold means minimum coverage to pass tests [OK]

Common Mistakes:

• Thinking threshold speeds up tests
• Believing threshold skips tests
• Assuming threshold hides reports

2. Which is the correct pytest command to set a coverage fail threshold at 80%?

easy

A. pytest --cov-fail-under=80

B. pytest --fail-under-cov=80

C. pytest --coverage-threshold=80

D. pytest --fail-coverage=80

Solution

1. Step 1: Recall pytest coverage threshold syntax

   The correct option uses --cov-fail-under to set minimum coverage.

2. Step 2: Match the correct command

   pytest --cov-fail-under=80 matches the exact pytest CLI option for coverage fail threshold.

3. Final Answer:

   pytest --cov-fail-under=80 -> Option A

4. Quick Check:

   Correct CLI option = --cov-fail-under [OK]

Hint: Use --cov-fail-under to set coverage threshold [OK]

Common Mistakes:

• Mixing option order or names
• Using non-existent flags
• Confusing coverage threshold with other options

3. Given this pytest command:
pytest --cov=myapp --cov-fail-under=90
and coverage report shows 85%, what will happen?

medium

A. Tests pass but with a warning

B. Tests fail due to coverage below 90%

C. Tests run without coverage check

D. Coverage report is not generated

Solution

1. Step 1: Understand the coverage threshold set

   The command sets a fail threshold at 90% coverage.

2. Step 2: Compare actual coverage with threshold

   Actual coverage is 85%, which is below 90%, so pytest fails the test run.

3. Final Answer:

   Tests fail due to coverage below 90% -> Option B

4. Quick Check:

   Coverage 85% < 90% threshold = fail [OK]

Hint: Coverage below threshold causes test failure [OK]

Common Mistakes:

• Assuming tests pass with warning
• Thinking coverage check is skipped
• Believing report is not generated

4. You set --cov-fail-under=75 but pytest does not fail even when coverage is 70%. What is the likely cause?

medium

A. You must use --fail-under-cov instead

B. Threshold value must be above 80%

C. Coverage is always ignored in pytest

D. Coverage plugin is not installed or enabled

Solution

1. Step 1: Check coverage plugin status

   If coverage plugin is missing or disabled, threshold has no effect.

2. Step 2: Understand threshold behavior

   Threshold works only if coverage plugin runs and measures coverage.

3. Final Answer:

   Coverage plugin is not installed or enabled -> Option D

4. Quick Check:

   Missing plugin = threshold ignored [OK]

Hint: Ensure coverage plugin is active for thresholds to work [OK]

Common Mistakes:

• Thinking threshold values have minimum limits
• Believing coverage is ignored by default
• Using wrong command option names

5. You want to enforce different coverage thresholds for branches and statements: 80% for branches and 90% for statements. Which pytest configuration snippet achieves this?

hard

A. [run]\nbranch = True\n[report]\nfail_under = 90\nfail_under_branch = 80

B. [run]\nbranch = True\n[report]\nfail_under = 80\nfail_under_branch = 90

C. [coverage]\nbranch = True\nthresholds = statements:90, branches:80

D. [coverage]\nfail_under_statements = 90\nfail_under_branches = 80

Solution

1. Step 1: Enable branch coverage in config

   Set branch = True under [run] to measure branch coverage.

2. Step 2: Set fail thresholds correctly

   Under [report], fail_under sets statement threshold, fail_under_branch sets branch threshold.

3. Final Answer:

   [run]\nbranch = True\n[report]\nfail_under = 90\nfail_under_branch = 80 -> Option A

4. Quick Check:

   Branch coverage enabled + correct thresholds = [run]\nbranch = True\n[report]\nfail_under = 90\nfail_under_branch = 80 [OK]

Hint: Use [run] branch=True and [report] fail_under_branch for branches [OK]

Common Mistakes:

• Swapping statement and branch thresholds
• Using wrong config section names
• Missing branch = True setting