Coverage in CI pipelines in PyTest - Build an Automation Script

Verify test coverage report generation in CI pipeline using pytest

Preconditions (3)

Step 1: Run pytest with coverage option in the CI pipeline

Step 2: Generate coverage report in XML format

Step 3: Upload or save the coverage report as an artifact

Step 4: Verify the coverage report file exists after the test run

✅ Expected Result: The coverage report XML file is generated and saved successfully in the CI pipeline after running pytest tests

Automation Requirements - pytest

Assertions Needed:

Assert that the coverage XML report file exists after test execution

Best Practices:

Use pytest-cov plugin for coverage measurement

Use explicit command line options to generate coverage report

Check for coverage report file existence as proof of coverage generation

Keep test code clean and modular

Automated Solution

PyTest

import os
import subprocess
import pytest


def test_coverage_report_generation():
    # Run pytest with coverage options to generate XML report
    result = subprocess.run([
        'pytest',
        '--cov=.',
        '--cov-report=xml:coverage.xml',
        '--maxfail=1',
        '--disable-warnings'
    ], capture_output=True, text=True)

    # Assert pytest run was successful
    assert result.returncode == 0, f"Pytest failed:\n{result.stdout}\n{result.stderr}"

    # Assert coverage.xml file is created
    assert os.path.exists('coverage.xml'), "Coverage XML report was not generated"

    # Clean up coverage file after test
    os.remove('coverage.xml')

if __name__ == '__main__':
    pytest.main([__file__])

This test runs pytest as a subprocess with coverage options to generate an XML report named coverage.xml.

We capture the output and check the return code to ensure tests passed.

Then we assert that the coverage.xml file exists, confirming coverage data was generated.

Finally, we clean up the coverage file to keep the environment clean.

This simulates what a CI pipeline would do: run tests with coverage and produce a report file.

Common Mistakes - 3 Pitfalls

Not using the pytest-cov plugin and trying to measure coverage manually

Not checking the pytest return code before asserting coverage file existence

Hardcoding file paths without cleanup

Bonus Challenge

Now add data-driven testing to run coverage report generation on multiple test folders

Show Hint