Coverage thresholds in PyTest - Framework Patterns

Framework Mode - Coverage thresholds

Folder Structure

tests/
├── test_example.py
├── test_utils.py
conftest.py
pytest.ini
requirements.txt

# Coverage reports output folder
coverage_reports/

Test Framework Layers

• Tests: Located in tests/ folder, contains test scripts like test_example.py.
• Fixtures & Setup: conftest.py holds shared setup code and fixtures for tests.
• Configuration: pytest.ini configures pytest and coverage plugin settings including coverage thresholds.
• Utilities: Helper functions or modules can be added as needed, imported by tests.
• Coverage Reports: Generated reports stored in coverage_reports/ folder for review.

Configuration Patterns

Use pytest.ini to configure coverage thresholds and reporting:

[pytest]
addopts = --cov=src --cov-report=term-missing --cov-fail-under=80

# Explanation:
# --cov=src : measure coverage for source code in src folder
# --cov-report=term-missing : show missing lines in terminal
# --cov-fail-under=80 : fail tests if coverage is below 80%

Adjust --cov-fail-under value to set your minimum coverage threshold.

For multiple environments or browsers, use environment variables or separate config files and load them in conftest.py.

Test Reporting and CI/CD Integration

• Coverage reports are generated automatically during test runs.
• Use --cov-report=xml option to create XML reports for CI tools.
• Integrate with CI/CD pipelines (GitHub Actions, Jenkins, GitLab CI) to run tests and enforce coverage thresholds.
• If coverage is below threshold, CI job fails, preventing merge/deployment.
• Store coverage reports as artifacts or publish them to dashboards for team visibility.

Best Practices

1. Set realistic coverage thresholds: Start with a reasonable percentage (e.g., 70-80%) and increase gradually.
2. Fail fast on low coverage: Use --cov-fail-under to prevent low-quality code from merging.
3. Use clear reports: Enable missing line reports to help developers find uncovered code easily.
4. Separate config from code: Keep coverage settings in pytest.ini or environment variables for easy updates.
5. Integrate with CI/CD: Automate coverage checks to maintain code quality continuously.

Self Check

Where in this folder structure would you add a new coverage threshold setting for a different environment?

Key Result

Use pytest.ini to set coverage thresholds that fail tests if coverage is below a set percentage, integrated with CI for quality enforcement.

Practice

1. What does setting a coverage threshold in pytest ensure?

easy

A. Only tests with coverage above 90% are executed

B. Tests run faster by skipping some checks

C. The test run fails if coverage is below a set percentage

D. Coverage reports are hidden from the user

Solution

1. Step 1: Understand coverage threshold purpose

   Coverage thresholds set a minimum coverage percentage to maintain test quality.

2. Step 2: Effect of coverage below threshold

   If coverage is below the threshold, pytest fails the test run to alert missing tests.

3. Final Answer:

   The test run fails if coverage is below a set percentage -> Option C

4. Quick Check:

   Coverage threshold = fail if below limit [OK]

Hint: Coverage threshold means minimum coverage to pass tests [OK]

Common Mistakes:

• Thinking threshold speeds up tests
• Believing threshold skips tests
• Assuming threshold hides reports

2. Which is the correct pytest command to set a coverage fail threshold at 80%?

easy

A. pytest --cov-fail-under=80

B. pytest --fail-under-cov=80

C. pytest --coverage-threshold=80

D. pytest --fail-coverage=80

Solution

1. Step 1: Recall pytest coverage threshold syntax

   The correct option uses --cov-fail-under to set minimum coverage.

2. Step 2: Match the correct command

   pytest --cov-fail-under=80 matches the exact pytest CLI option for coverage fail threshold.

3. Final Answer:

   pytest --cov-fail-under=80 -> Option A

4. Quick Check:

   Correct CLI option = --cov-fail-under [OK]

Hint: Use --cov-fail-under to set coverage threshold [OK]

Common Mistakes:

• Mixing option order or names
• Using non-existent flags
• Confusing coverage threshold with other options

3. Given this pytest command:
pytest --cov=myapp --cov-fail-under=90
and coverage report shows 85%, what will happen?

medium

A. Tests pass but with a warning

B. Tests fail due to coverage below 90%

C. Tests run without coverage check

D. Coverage report is not generated

Solution

1. Step 1: Understand the coverage threshold set

   The command sets a fail threshold at 90% coverage.

2. Step 2: Compare actual coverage with threshold

   Actual coverage is 85%, which is below 90%, so pytest fails the test run.

3. Final Answer:

   Tests fail due to coverage below 90% -> Option B

4. Quick Check:

   Coverage 85% < 90% threshold = fail [OK]

Hint: Coverage below threshold causes test failure [OK]

Common Mistakes:

• Assuming tests pass with warning
• Thinking coverage check is skipped
• Believing report is not generated

4. You set --cov-fail-under=75 but pytest does not fail even when coverage is 70%. What is the likely cause?

medium

A. You must use --fail-under-cov instead

B. Threshold value must be above 80%

C. Coverage is always ignored in pytest

D. Coverage plugin is not installed or enabled

Solution

1. Step 1: Check coverage plugin status

   If coverage plugin is missing or disabled, threshold has no effect.

2. Step 2: Understand threshold behavior

   Threshold works only if coverage plugin runs and measures coverage.

3. Final Answer:

   Coverage plugin is not installed or enabled -> Option D

4. Quick Check:

   Missing plugin = threshold ignored [OK]

Hint: Ensure coverage plugin is active for thresholds to work [OK]

Common Mistakes:

• Thinking threshold values have minimum limits
• Believing coverage is ignored by default
• Using wrong command option names

5. You want to enforce different coverage thresholds for branches and statements: 80% for branches and 90% for statements. Which pytest configuration snippet achieves this?

hard

A. [run]\nbranch = True\n[report]\nfail_under = 90\nfail_under_branch = 80

B. [run]\nbranch = True\n[report]\nfail_under = 80\nfail_under_branch = 90

C. [coverage]\nbranch = True\nthresholds = statements:90, branches:80

D. [coverage]\nfail_under_statements = 90\nfail_under_branches = 80

Solution

1. Step 1: Enable branch coverage in config

   Set branch = True under [run] to measure branch coverage.

2. Step 2: Set fail thresholds correctly

   Under [report], fail_under sets statement threshold, fail_under_branch sets branch threshold.

3. Final Answer:

   [run]\nbranch = True\n[report]\nfail_under = 90\nfail_under_branch = 80 -> Option A

4. Quick Check:

   Branch coverage enabled + correct thresholds = [run]\nbranch = True\n[report]\nfail_under = 90\nfail_under_branch = 80 [OK]

Hint: Use [run] branch=True and [report] fail_under_branch for branches [OK]

Common Mistakes:

• Swapping statement and branch thresholds
• Using wrong config section names
• Missing branch = True setting