Report publishing in CI in Selenium Java - Build an Automation Script

Verify test report is generated and published in CI pipeline

Preconditions (3)

Step 1: Trigger the CI pipeline to run the Selenium test suite

Step 2: Wait for the test execution to complete

Step 3: Locate the generated test report file (e.g., target/surefire-reports/index.html)

Step 4: Verify the report file exists and is not empty

Step 5: Check the CI pipeline logs or artifacts section to confirm the report is published and accessible

✅ Expected Result: The Selenium test suite runs successfully in CI, the test report file is generated and published as a pipeline artifact, and it can be accessed from the CI interface.

Automation Requirements - Selenium with TestNG and Maven

Assertions Needed:

Assert test report file exists after test execution

Assert test report file size is greater than zero

Best Practices:

Use TestNG annotations for setup and teardown

Use Maven Surefire plugin for report generation

Use explicit waits in Selenium tests

Keep test code and report generation separate

Ensure reports are archived as CI artifacts

Automated Solution

Selenium Java

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.Assert;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

import java.io.File;
import java.time.Duration;

public class ReportPublishingTest {
    private WebDriver driver;

    @BeforeClass
    public void setUp() {
        // Set path to chromedriver executable if needed
        // System.setProperty("webdriver.chrome.driver", "/path/to/chromedriver");
        driver = new ChromeDriver();
        driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(10));
    }

    @Test
    public void sampleTest() {
        driver.get("https://example.com");
        String title = driver.getTitle();
        Assert.assertTrue(title.contains("Example Domain"), "Title should contain 'Example Domain'");
    }

    @AfterClass
    public void tearDown() {
        if (driver != null) {
            driver.quit();
        }
    }

    // This method simulates verification of report file after tests run
    @Test(dependsOnMethods = {"sampleTest"})
    public void verifyReportFile() {
        // Path to Maven Surefire report file
        File reportFile = new File("target/surefire-reports/index.html");
        Assert.assertTrue(reportFile.exists(), "Test report file should exist");
        Assert.assertTrue(reportFile.length() > 0, "Test report file should not be empty");
    }
}

The setUp method initializes the Chrome WebDriver with an implicit wait to handle element loading.

The sampleTest method navigates to example.com and asserts the page title contains 'Example Domain' to simulate a passing test.

The tearDown method closes the browser after tests complete.

The verifyReportFile test runs after the sample test and checks if the Maven Surefire report file target/surefire-reports/index.html exists and is not empty, ensuring the report was generated.

In a real CI environment, the pipeline should archive this report file as an artifact for publishing. This test confirms the report generation step.

Common Mistakes - 4 Pitfalls

Not waiting for tests to complete before checking report file

Hardcoding absolute paths for report files

Not archiving or publishing the report as a CI artifact

Using Thread.sleep instead of implicit or explicit waits in Selenium tests

Bonus Challenge

Now add data-driven testing with 3 different URLs and verify their titles

Show Hint

Practice

1. What is the main purpose of publishing test reports in a Continuous Integration (CI) pipeline?

easy

A. To increase the speed of test execution

B. To automatically fix failed tests

C. To reduce the number of test cases

D. To quickly show test results after code changes

Solution

1. Step 1: Understand CI pipeline goals

   CI pipelines aim to integrate code changes frequently and provide feedback quickly.

2. Step 2: Role of test reports in CI

   Publishing test reports helps teams see if tests passed or failed right after changes.

3. Final Answer:

   To quickly show test results after code changes -> Option D

4. Quick Check:

   Test reports in CI = Quick feedback [OK]

Hint: Test reports in CI = fast feedback on code changes [OK]

Common Mistakes:

• Confusing report publishing with speeding up tests
• Thinking reports reduce test cases
• Believing reports fix tests automatically

2. Which of the following is the correct way to configure a Jenkins pipeline to publish JUnit test reports generated by Selenium Java tests?

easy

A. Use the junit '**/target/surefire-reports/*.xml' step in the pipeline script

B. Add publishJUnitReports() in the Java test code

C. Call driver.publishReports() in Selenium test methods

D. Use archiveArtifacts('**/*.html') to publish XML reports

Solution

1. Step 1: Identify Jenkins syntax for JUnit reports

   Jenkins uses the junit step with a file pattern to locate XML reports.

2. Step 2: Match correct usage

   junit '**/target/surefire-reports/*.xml' correctly points to JUnit XML files generated by Maven Surefire plugin.

3. Final Answer:

   Use the junit '**/target/surefire-reports/*.xml' step in the pipeline script -> Option A

4. Quick Check:

   Jenkins pipeline uses junit step with XML path [OK]

Hint: Jenkins publishes JUnit XML with junit('path') step [OK]

Common Mistakes:

• Trying to publish reports from Java code directly
• Using wrong Jenkins steps for report publishing
• Archiving HTML instead of XML for JUnit reports

3. Given the following Jenkins pipeline snippet, what will be the result of the junit '**/target/surefire-reports/*.xml' step if no XML files are found?

pipeline {
  agent any
  stages {
    stage('Test') {
      steps {
        sh 'mvn test'
        junit '**/target/surefire-reports/*.xml'
      }
    }
  }
}

medium

A. The build will fail with an error about missing reports

B. The build will pass with no test results recorded

C. The build will retry the test stage automatically

D. The build will pass but mark tests as skipped

Solution

1. Step 1: Understand junit step behavior with missing files

   If no XML files match, Jenkins does not fail the build but records no test results.

2. Step 2: Confirm build status

   Since no test results are found, the build passes but shows zero tests executed.

3. Final Answer:

   The build will pass with no test results recorded -> Option B

4. Quick Check:

   No XML files = build passes, no tests recorded [OK]

Hint: No XML files found means no tests recorded, build still passes [OK]

Common Mistakes:

• Assuming build fails if reports missing
• Thinking tests are marked skipped automatically
• Expecting automatic retries on missing reports

4. You configured your CI pipeline to publish Selenium test reports, but the reports are not visible in the CI dashboard. Which of the following is the most likely cause?

medium

A. The CI server does not support publishing reports

B. The Selenium tests did not run any assertions

C. The test report files are generated in a different directory than specified in the CI config

D. The test code uses deprecated Selenium APIs

Solution

1. Step 1: Check report file paths in CI config

   If the CI pipeline looks for reports in the wrong folder, it won't find or display them.

2. Step 2: Verify report generation location

   Ensure test reports are generated where the CI expects them, matching the configured path.

3. Final Answer:

   The test report files are generated in a different directory than specified in the CI config -> Option C

4. Quick Check:

   Report path mismatch = no reports shown [OK]

Hint: Check report file paths match between test and CI config [OK]

Common Mistakes:

• Blaming test assertions for missing reports
• Assuming CI server cannot publish reports
• Thinking deprecated APIs affect report visibility

5. You want to publish Selenium Java test reports in a CI pipeline that uses GitHub Actions. Which combination of steps correctly collects and publishes the test results?

hard

A. Run tests with Maven, then use actions/upload-artifact to save reports, and configure actions/junit-report to display them

B. Run tests with Maven, then use actions/download-artifact to save reports, and actions/junit-report to display them

C. Run tests with Maven, then use actions/upload-artifact to save reports, and actions/upload-test-report to display them

D. Run tests with Maven, then use actions/upload-artifact to save reports, and no further steps are needed to display reports

Solution

1. Step 1: Save test reports as artifacts

   Use actions/upload-artifact to store test report files after running tests.

2. Step 2: Use a report action to display results

   Configure actions/junit-report or similar to parse and show test results in GitHub UI.

3. Final Answer:

   Run tests with Maven, then use actions/upload-artifact to save reports, and configure actions/junit-report to display them -> Option A

4. Quick Check:

   Upload artifact + report action = published test results [OK]

Hint: Upload reports artifact then use report action to display [OK]

Common Mistakes:

• Using upload-test-report action which does not exist
• Downloading artifacts instead of uploading
• Skipping report display step after upload