Test execution reporting in Testing Fundamentals - Framework Patterns

Framework Mode - Test execution reporting

Folder Structure for Test Execution Reporting

  test-project/
  ├── reports/               # Stores test execution reports
  │   ├── html/              # HTML format reports for easy viewing
  │   ├── json/              # JSON format reports for integrations
  │   └── logs/              # Raw logs from test runs
  ├── tests/                 # Test scripts
  ├── utils/                 # Helper functions for reporting and logging
  ├── config/                # Configuration files for environments and reporting
  └── ci/                    # CI/CD pipeline scripts and configs
  

Test Framework Layers for Reporting

• Test Scripts Layer: Contains the actual test cases that generate results.
• Reporting Layer: Collects test results, formats them, and saves reports (HTML, JSON, logs).
• Utilities Layer: Provides helper functions for logging, screenshots, and report generation.
• Configuration Layer: Holds settings for report formats, output locations, and notification preferences.
• CI/CD Integration Layer: Connects test runs and reports with automated pipelines for continuous feedback.

Configuration Patterns for Test Execution Reporting

• Environment Settings: Define where reports are saved and which formats to generate (e.g., HTML, JSON).
• Report Naming: Use timestamps or build numbers to keep reports organized and unique.
• Notification Settings: Configure email or messaging alerts with report summaries after test runs.
• Log Levels: Set verbosity for logs (info, warning, error) to control report detail.
• CI/CD Variables: Use environment variables to adjust reporting behavior depending on pipeline stage.

Test Reporting and CI/CD Integration

• Report Generation: Automatically create readable reports after tests finish, showing pass/fail and details.
• Dashboard Integration: Upload reports to dashboards for team visibility.
• Alerts and Notifications: Send summaries or failures to team channels (email, Slack).
• CI/CD Pipeline: Run tests and generate reports as part of automated build and deployment pipelines.
• Historical Data: Store reports to track test trends over time.

Best Practices for Test Execution Reporting

1. Clear and Simple Reports: Use easy-to-read formats with clear pass/fail status and error messages.
2. Consistent Naming and Storage: Organize reports by date and test run to avoid confusion.
3. Automate Report Generation: Integrate reporting into test runs so no manual steps are needed.
4. Include Logs and Screenshots: Attach helpful details to reports for faster debugging.
5. Integrate with Team Tools: Share reports automatically with the team through chat or email.

Self Check Question

Where in this folder structure would you add a new utility to generate PDF test execution reports?

Key Result

Organize test execution reporting with clear folder structure, layered design, configurable settings, and CI/CD integration for automated, readable reports.

Practice

1. What is the main purpose of a test execution report in software testing?

easy

A. To write new test cases

B. To design the software architecture

C. To show which tests passed or failed

D. To deploy the software to users

Solution

1. Step 1: Understand the role of test execution reports

   Test execution reports summarize the results of running tests, showing which tests passed or failed.

2. Step 2: Eliminate unrelated options

   Writing test cases, designing software, and deploying are different activities not related to reporting test results.

3. Final Answer:

   To show which tests passed or failed -> Option C

4. Quick Check:

   Test execution report = pass/fail summary [OK]

Hint: Reports show pass/fail status clearly [OK]

Common Mistakes:

• Confusing report with test design
• Thinking reports deploy software
• Assuming reports write tests

2. Which of the following is a common element included in a test execution report?

easy

A. Test case names and their status

B. Source code of the application

C. User interface design sketches

D. Database schema diagrams

Solution

1. Step 1: Identify typical report contents

   Test execution reports usually list test case names and whether they passed or failed.

2. Step 2: Remove unrelated technical documents

   Source code, UI sketches, and database diagrams are not part of test execution reports.

3. Final Answer:

   Test case names and their status -> Option A

4. Quick Check:

   Report = test names + status [OK]

Hint: Reports list tests and results, not design files [OK]

Common Mistakes:

• Including source code in reports
• Confusing reports with design documents
• Expecting diagrams in test reports

3. Consider this test execution report snippet:
TestLogin: Passed
TestPayment: Failed
TestLogout: Passed
How many tests passed according to this report?

medium

A. 1

B. 2

C. 3

D. 0

Solution

1. Step 1: Count tests marked as Passed

   TestLogin and TestLogout are marked Passed, so 2 tests passed.

2. Step 2: Confirm no other Passed tests

   TestPayment is Failed, so it does not count.

3. Final Answer:

   2 -> Option B

4. Quick Check:

   Passed tests count = 2 [OK]

Hint: Count only 'Passed' labels in report [OK]

Common Mistakes:

• Counting failed tests as passed
• Misreading the report lines
• Counting total tests instead of passed

4. A test execution report shows:
TestA: Passed
TestB: Passed
TestC: Passed
TestD: Passed
But the tester says TestC actually failed. What is the most likely problem?

medium

A. TestD result was copied to TestC by mistake

B. TestC was never executed

C. All tests actually passed

D. The report was not updated after the test run

Solution

1. Step 1: Understand report and tester mismatch

   If the report shows TestC passed but tester says it failed, the report is outdated or incorrect.

2. Step 2: Evaluate other options

   TestC was likely executed but result not updated; copying results is less common; all passing contradicts tester.

3. Final Answer:

   The report was not updated after the test run -> Option D

4. Quick Check:

   Report outdated = wrong pass shown [OK]

Hint: Mismatch means report not refreshed [OK]

Common Mistakes:

• Assuming test was not run
• Believing all tests passed
• Ignoring report update timing

5. You have a test execution report with 10 tests: 7 passed, 2 failed, and 1 skipped. How should this report be presented to best help the development team?

hard

A. List all tests with status, highlight failed tests, and include error details

B. Show only passed tests to keep the report short

C. Exclude skipped tests and failed tests from the report

D. Report only the total number of tests run without details

Solution

1. Step 1: Identify useful report content

   Listing all tests with their status helps the team see the full picture.

2. Step 2: Emphasize failed tests and errors

   Highlighting failures and showing error details helps developers fix issues quickly.

3. Final Answer:

   List all tests with status, highlight failed tests, and include error details -> Option A

4. Quick Check:

   Complete, clear report = best for fixing bugs [OK]

Hint: Show all results, focus on failures with details [OK]

Common Mistakes:

• Hiding failed or skipped tests
• Giving only totals without details
• Ignoring error messages