Running PyTest in Jenkins - Build an Automation Script

Run PyTest tests in Jenkins and verify test results

Preconditions (4)

Step 1: Open Jenkins dashboard

Step 2: Navigate to the Jenkins job for the project

Step 3: Configure the job to run a shell command: 'pytest --junitxml=results.xml'

Step 4: Add a post-build action to publish JUnit test results using 'results.xml'

Step 5: Save the job configuration

Step 6: Click 'Build Now' to run the job

Step 7: Wait for the build to complete

Step 8: Open the build details page

Step 9: Verify that the test results are displayed with passed and failed tests

✅ Expected Result: The Jenkins build runs PyTest tests, generates a JUnit XML report, and Jenkins displays the test results correctly showing which tests passed or failed.

Automation Requirements - pytest

Assertions Needed:

Verify the Jenkins build completes successfully

Verify the JUnit XML report file 'results.xml' is generated

Verify the test results contain expected passed and failed test counts

Best Practices:

Use explicit commands to run PyTest with JUnit XML output

Use Jenkins post-build actions to publish test reports

Keep test commands and report paths consistent

Use Jenkins pipeline or freestyle job with clear steps

Automated Solution

PyTest

import os
import subprocess
import xml.etree.ElementTree as ET
import unittest

class TestJenkinsPytestRun(unittest.TestCase):
    def test_pytest_run_and_report(self):
        # Run pytest with JUnit XML output
        result = subprocess.run(['pytest', '--junitxml=results.xml'], capture_output=True, text=True)
        # Assert pytest command ran successfully
        self.assertEqual(result.returncode, 0, f"Pytest failed: {result.stderr}")
        # Check if results.xml file exists
        self.assertTrue(os.path.exists('results.xml'), 'JUnit XML report not found')
        # Parse the XML report
        tree = ET.parse('results.xml')
        root = tree.getroot()
        # Extract test counts
        tests = int(root.attrib.get('tests', 0))
        failures = int(root.attrib.get('failures', 0))
        errors = int(root.attrib.get('errors', 0))
        skipped = int(root.attrib.get('skipped', 0))
        # Assert there is at least one test run
        self.assertGreater(tests, 0, 'No tests were run')
        # Assert no errors in test run
        self.assertEqual(errors, 0, 'There were errors in the test run')
        # Optionally assert failures count (can be zero or more)
        # Print summary
        print(f"Tests run: {tests}, Failures: {failures}, Skipped: {skipped}")

if __name__ == '__main__':
    unittest.main()

This script automates running PyTest with the --junitxml=results.xml option to generate a JUnit XML report file.

It uses Python's subprocess module to run the PyTest command and captures the output and return code to verify the test run succeeded.

Then it checks if the results.xml file was created, which Jenkins uses to display test results.

The script parses the XML report using xml.etree.ElementTree to extract the total tests, failures, errors, and skipped counts.

Assertions verify that tests were run, no errors occurred, and the report file exists.

This matches what Jenkins expects when configured to run PyTest and publish JUnit reports, ensuring the automation covers the manual test case steps.

Common Mistakes - 4 Pitfalls

{'mistake': 'Not generating the JUnit XML report with PyTest', 'why_bad': 'Jenkins cannot display test results without the XML report file.', 'correct_approach': "Always run PyTest with the '--junitxml=filename.xml' option to generate the report."}

Hardcoding file paths without considering Jenkins workspace

Ignoring PyTest return codes and errors

{'mistake': 'Not configuring Jenkins post-build action to publish test results', 'why_bad': "Test results won't be visible in Jenkins UI without publishing the report.", 'correct_approach': "Add 'Publish JUnit test result report' post-build action pointing to the XML report."}

Bonus Challenge

Now add data-driven testing with 3 different test input scenarios in PyTest and verify Jenkins runs all scenarios.

Show Hint

Practice

1. What is the main purpose of running pytest in Jenkins?

easy

A. To monitor server performance during tests

B. To deploy Python applications automatically

C. To write new Python test cases inside Jenkins

D. To automate running Python tests and see results in Jenkins

Solution

1. Step 1: Understand Jenkins role in testing

   Jenkins automates tasks like running tests without manual effort.

2. Step 2: Identify pytest's role

   Pytest runs Python tests and generates results for Jenkins to display.

3. Final Answer:

   To automate running Python tests and see results in Jenkins -> Option D

4. Quick Check:

   Automation of tests = A [OK]

Hint: Jenkins runs tests automatically to show results [OK]

Common Mistakes:

• Confusing test running with deployment
• Thinking Jenkins writes tests
• Assuming Jenkins monitors server performance

2. Which Jenkins Pipeline command correctly runs pytest and saves results in XML format?

easy

A. sh 'pytest --junitxml=results.xml'

B. sh 'pytest -o junit=results.xml'

C. sh 'pytest --xml=results.xml'

D. sh 'pytest --save=results.xml'

Solution

1. Step 1: Recall pytest XML output option

   The correct option to save results in XML is --junitxml=filename.

2. Step 2: Match Jenkins shell command syntax

   Jenkins uses sh to run shell commands, so sh 'pytest --junitxml=results.xml' is correct.

3. Final Answer:

   sh 'pytest --junitxml=results.xml' -> Option A

4. Quick Check:

   Correct pytest XML flag = D [OK]

Hint: Use --junitxml=filename to save pytest results [OK]

Common Mistakes:

• Using wrong pytest flags for XML output
• Incorrect Jenkins shell command syntax
• Confusing XML output with other formats

3. Given this Jenkins Pipeline snippet:

pipeline {
  agent any
  stages {
    stage('Test') {
      steps {
        sh 'pytest --junitxml=results.xml'
        junit 'results.xml'
      }
    }
  }
}

What will Jenkins display after running this pipeline?

medium

A. Test results summary with passed and failed tests

B. Only console output without test results

C. An error because junit step is missing

D. No output because tests are not run

Solution

1. Step 1: Understand the sh step

   The sh command runs pytest and generates results.xml with test results.

2. Step 2: Understand the junit step

   The junit step reads results.xml and shows test results in Jenkins UI.

3. Final Answer:

   Test results summary with passed and failed tests -> Option A

4. Quick Check:

   pytest + junit step = test summary shown [OK]

Hint: Use junit step to publish pytest XML results [OK]

Common Mistakes:

• Forgetting to add junit step to publish results
• Assuming pytest output alone shows results in Jenkins
• Confusing console logs with test reports

4. You run this Jenkins Pipeline:

sh 'pytest --junitxml=results.xml'
junit 'results.xml'

But Jenkins shows an error: FileNotFoundError: results.xml not found. What is the likely cause?

medium

A. Jenkins workspace is missing

B. Pytest did not run or failed before creating results.xml

C. The junit step is misspelled

D. The XML file is created but in a different directory

Solution

1. Step 1: Check if pytest ran successfully

   If pytest fails or does not run, it won't create results.xml.

2. Step 2: Confirm file existence before junit step

   The junit step needs the XML file; if missing, it errors out.

3. Final Answer:

   Pytest did not run or failed before creating results.xml -> Option B

4. Quick Check:

   Missing XML means pytest didn't create it [OK]

Hint: Check pytest success before junit step [OK]

Common Mistakes:

• Assuming junit step spelling causes file error
• Ignoring pytest failure logs
• Not verifying file path correctness

5. You want Jenkins to run pytest only on files changed in a Git branch and publish results. Which approach best fits this requirement?

hard

A. Manually run pytest locally and upload results to Jenkins

B. Run pytest on all files every time and ignore changed files

C. Use Jenkins Pipeline to detect changed files, run pytest on those, then publish with junit

D. Use Jenkins to deploy code without running tests

Solution

1. Step 1: Detect changed files in Jenkins Pipeline

   Use Git commands or plugins to find changed Python files in the branch.

2. Step 2: Run pytest only on those changed files and publish results

   Run pytest with paths of changed files, then use junit to show results.

3. Final Answer:

   Use Jenkins Pipeline to detect changed files, run pytest on those, then publish with junit -> Option C

4. Quick Check:

   Selective test run + publish = A [OK]

Hint: Detect changed files, run pytest on them, publish results [OK]

Common Mistakes:

• Running all tests every time wasting resources
• Skipping automated test runs
• Not publishing test results in Jenkins