Why coverage measures test completeness in PyTest - Framework Benefits

Framework Mode - Why coverage measures test completeness

Folder Structure of a Pytest Project with Coverage

project-root/
├── src/
│   └── app.py
├── tests/
│   ├── test_app.py
│   └── __init__.py
├── .coveragerc
├── pytest.ini
├── requirements.txt
└── README.md
  

Test Framework Layers

• Source Code Layer: The application code inside src/.
• Test Layer: Test cases inside tests/ using pytest functions.
• Coverage Layer: Coverage tool measures which parts of source code are run by tests.
• Configuration Layer: Files like .coveragerc and pytest.ini to set coverage options and test settings.
• Utilities Layer: Fixtures or helper functions inside tests/ or separate utility modules.

Configuration Patterns for Coverage in Pytest

Use a .coveragerc file to specify coverage settings:

[run]
source = src
branch = True

[report]
show_missing = True
skip_covered = False
  

Use pytest.ini to integrate coverage plugin:

[pytest]
addopts = --cov=src --cov-report=term-missing
  

This setup runs tests and shows which lines of src/ are tested or missed.

Test Reporting and CI/CD Integration

• Coverage reports show percentage of code tested (line and branch coverage).
• Reports can be in terminal, HTML, XML formats for easy reading.
• CI/CD pipelines run pytest --cov to check coverage on every code change.
• Fail builds if coverage drops below a set threshold to keep tests complete.

Best Practices for Using Coverage to Measure Test Completeness

• Measure both line and branch coverage for better completeness insight.
• Configure coverage to focus on source code, excluding tests and external libraries.
• Use coverage reports to find untested code and write new tests accordingly.
• Integrate coverage checks into CI to maintain test quality over time.
• Remember coverage shows what code runs, but not if tests check correct behavior--combine with assertions.

Self Check Question

Where in this folder structure would you add a new test file to increase coverage for a new feature?

Key Result

Coverage tools measure which parts of the source code are exercised by tests, helping ensure test completeness.

Practice

1. What does test coverage measure in pytest?

easy

A. How much of the code is executed by tests

B. How many tests are written

C. How fast the tests run

D. How many errors tests find

Solution

1. Step 1: Understand the meaning of coverage

   Coverage shows which parts of the code are run when tests execute.

2. Step 2: Compare options to coverage definition

   Only How much of the code is executed by tests matches this meaning, others describe different test aspects.

3. Final Answer:

   How much of the code is executed by tests -> Option A

4. Quick Check:

   Coverage = executed code percentage [OK]

Hint: Coverage = code run by tests, not test count [OK]

Common Mistakes:

• Confusing coverage with number of tests
• Thinking coverage measures test speed
• Believing coverage counts errors found

2. Which pytest command correctly runs tests with coverage measurement?

easy

A. pytest --cover

B. pytest --coverage

C. pytest -cov-report

D. pytest --cov

Solution

1. Step 1: Recall pytest coverage plugin syntax

   The correct flag to measure coverage is '--cov'.

2. Step 2: Check options for correctness

   Only pytest --cov uses the exact correct flag '--cov'. Others are invalid or incomplete.

3. Final Answer:

   pytest --cov -> Option D

4. Quick Check:

   Use --cov to enable coverage [OK]

Hint: Use '--cov' flag to measure coverage in pytest [OK]

Common Mistakes:

• Using '--coverage' instead of '--cov'
• Mixing coverage report flags with coverage run flags
• Typing '--cover' which is invalid

3. Given this pytest coverage output:

Name          Stmts   Miss  Cover
my_module.py     10      2    80%

What does the 'Miss' number mean?

medium

A. Number of lines not executed by tests

B. Number of errors in code

C. Number of tests skipped

D. Number of tests that failed

Solution

1. Step 1: Understand coverage report columns

   'Miss' shows how many lines of code were not run by tests.

2. Step 2: Match 'Miss' meaning to options

   Number of lines not executed by tests correctly describes 'Miss' as unexecuted lines; others describe unrelated test results.

3. Final Answer:

   Number of lines not executed by tests -> Option A

4. Quick Check:

   Miss = untested lines count [OK]

Hint: 'Miss' means lines tests did not run [OK]

Common Mistakes:

• Thinking 'Miss' counts failed tests
• Confusing 'Miss' with skipped tests
• Assuming 'Miss' means code errors

4. You run pytest with coverage but get 0% coverage report. What is the most likely cause?

medium

A. Tests passed too quickly

B. Tests did not execute any code

C. Coverage plugin is not installed

D. Code has no functions

Solution

1. Step 1: Analyze 0% coverage meaning

   0% coverage means no code lines were run during tests.

2. Step 2: Evaluate causes

   If the coverage plugin is not installed, pytest will not measure coverage and may silently produce 0% coverage report or no coverage data.

3. Final Answer:

   Coverage plugin is not installed -> Option C

4. Quick Check:

   Missing plugin causes no coverage data [OK]

Hint: 0% coverage often means coverage plugin missing [OK]

Common Mistakes:

• Assuming plugin missing causes 0% without errors
• Thinking fast tests mean low coverage
• Believing code without functions can't be covered

5. You want to improve test completeness using coverage. Which approach is best?

hard

A. Write more tests without checking coverage

B. Add tests targeting uncovered code lines shown by coverage report

C. Ignore coverage and focus on test speed

D. Remove tests that run covered code

Solution

1. Step 1: Understand coverage report use

   Coverage shows which code lines lack tests, guiding where to add tests.

2. Step 2: Evaluate options for improving completeness

   Only Add tests targeting uncovered code lines shown by coverage report uses coverage data to add tests for uncovered code, improving completeness.

3. Final Answer:

   Add tests targeting uncovered code lines shown by coverage report -> Option B

4. Quick Check:

   Use coverage to find and test missing code [OK]

Hint: Add tests where coverage report shows gaps [OK]

Common Mistakes:

• Writing tests blindly without coverage info
• Ignoring coverage to focus on speed
• Removing tests that cover code