Risk-based testing in Testing Fundamentals - Framework Patterns

Framework Mode - Risk-based testing

Folder Structure for Risk-Based Testing Framework

  risk-based-testing-framework/
  ├── tests/                  # Test cases organized by risk priority
  │   ├── high_risk/          # Tests for high-risk features
  │   ├── medium_risk/        # Tests for medium-risk features
  │   └── low_risk/           # Tests for low-risk features
  ├── risk_analysis/          # Documents and scripts for risk assessment
  │   ├── risk_matrix.xlsx    # Risk matrix file
  │   └── risk_assessment.md  # Risk assessment notes
  ├── config/                 # Configuration files
  │   └── environment.yaml    # Environment and test settings
  ├── utils/                  # Utility functions and helpers
  │   └── risk_helpers.py     # Helpers for risk calculations
  ├── reports/                # Test execution reports
  └── README.md               # Project overview and instructions
  

Test Framework Layers in Risk-Based Testing

• Risk Analysis Layer: Contains risk assessment documents and tools to identify and prioritize risks.
• Test Design Layer: Organizes test cases based on risk levels (high, medium, low) to focus testing effort.
• Test Execution Layer: Runs tests with priority given to high-risk areas first.
• Utilities Layer: Provides helper functions for risk scoring, filtering tests by risk, and reporting.
• Configuration Layer: Manages environment settings, test data, and risk thresholds.

Configuration Patterns for Risk-Based Testing

• Risk Thresholds: Define risk score cutoffs in config files (e.g., environment.yaml) to classify tests into high, medium, or low risk.
• Environment Settings: Specify test environments and browsers to run tests against.
• Credentials Management: Store sensitive data securely and reference in tests without hardcoding.
• Test Selection: Use config flags to run only tests above a certain risk level for faster feedback.

Test Reporting and CI/CD Integration

• Risk-Based Test Reports: Reports highlight test results grouped by risk level to show coverage of critical areas.
• Trend Analysis: Track failures in high-risk areas over time to focus improvements.
• CI/CD Integration: Configure pipelines to run high-risk tests on every commit and full suite nightly.
• Alerts: Notify team immediately if high-risk tests fail to prevent critical issues.

Best Practices for Risk-Based Testing Framework

1. Prioritize Tests by Risk: Always focus first on tests covering the highest risk features to catch critical bugs early.
2. Keep Risk Assessment Updated: Regularly review and update risk analysis as the product changes.
3. Automate Risk Filtering: Use utilities to automatically select tests based on risk scores for flexible test runs.
4. Integrate with CI/CD: Ensure risk-based test runs are part of automated pipelines for fast feedback.
5. Clear Documentation: Maintain clear documentation of risk criteria and test organization for team alignment.

Self-Check Question

Where in this folder structure would you add a new test case for a feature identified as high risk?

Key Result

Organize and prioritize tests based on risk levels to focus testing effort on critical areas.

Practice

1. What is the main goal of risk-based testing in software testing?

easy

A. To avoid testing parts that are easy to test

B. To test all parts of the software equally

C. To focus testing on the most important and risky parts first

D. To test only the user interface

Solution

1. Step 1: Understand the purpose of risk-based testing

   Risk-based testing aims to prioritize testing efforts on areas with the highest risk to save time and improve effectiveness.

2. Step 2: Compare options with this purpose

   To focus testing on the most important and risky parts first matches this goal by focusing on important and risky parts first, while others do not.

3. Final Answer:

   To focus testing on the most important and risky parts first -> Option C

4. Quick Check:

   Risk-based testing = Focus on risky parts first [OK]

Hint: Risk-based testing targets risky parts first [OK]

Common Mistakes:

• Thinking all parts are tested equally
• Believing risk-based testing ignores important parts
• Confusing risk-based testing with UI-only testing

2. Which of the following is the correct way to prioritize tests in risk-based testing?

easy

A. By combining the impact and likelihood of potential problems

B. By testing the easiest features first

C. By testing features in alphabetical order

D. By testing only the newest features

Solution

1. Step 1: Recall how risk is calculated in testing

   Risk is usually calculated by combining the impact of a problem and how likely it is to happen.

2. Step 2: Match this with the options

   By combining the impact and likelihood of potential problems correctly states this combination, while others focus on unrelated criteria.

3. Final Answer:

   By combining the impact and likelihood of potential problems -> Option A

4. Quick Check:

   Risk = Impact x Likelihood [OK]

Hint: Prioritize tests by impact and likelihood [OK]

Common Mistakes:

• Prioritizing by ease or alphabetical order
• Ignoring likelihood of problems
• Testing only new features without risk analysis

3. Consider this simplified risk table for features:

Feature A: Impact=5, Likelihood=4
Feature B: Impact=3, Likelihood=6
Feature C: Impact=4, Likelihood=2

Which feature should be tested first based on risk?

medium

A. Feature B

B. Feature A

C. Feature C

D. All features equally

Solution

1. Step 1: Calculate risk for each feature

   Risk = Impact x Likelihood.
   Feature A: 5 x 4 = 20
   Feature B: 3 x 6 = 18
   Feature C: 4 x 2 = 8

2. Step 2: Identify highest risk

   Feature A has the highest risk score (20), so it should be tested first.

3. Final Answer:

   Feature A -> Option B

4. Quick Check:

   Highest risk = Feature A (20) [OK]

Hint: Multiply impact and likelihood to find highest risk [OK]

Common Mistakes:

• Choosing feature with highest likelihood only
• Ignoring impact in calculation
• Testing all features equally without prioritization

4. A tester wrote this pseudocode to select features to test based on risk:

features = [{"name": "A", "impact": 5, "likelihood": 4},
            {"name": "B", "impact": 3, "likelihood": 6}]

for feature in features:
    risk = feature["impact"] + feature["likelihood"]
    if risk > 8:
        print(feature["name"])

What is wrong with this code for risk-based testing?

medium

A. Risk should be calculated by multiplying impact and likelihood, not adding

B. The loop should use feature.name instead of feature["name"]

C. The risk threshold should be less than 8, not greater

D. The features list should be sorted before looping

Solution

1. Step 1: Review risk calculation method

   Risk-based testing calculates risk by multiplying impact and likelihood, not adding them.

2. Step 2: Identify error in code

   The code adds impact and likelihood, which is incorrect for risk calculation.

3. Final Answer:

   Risk should be calculated by multiplying impact and likelihood, not adding -> Option A

4. Quick Check:

   Risk = Impact x Likelihood, not sum [OK]

Hint: Risk = impact x likelihood, not sum [OK]

Common Mistakes:

• Adding instead of multiplying impact and likelihood
• Confusing dictionary access syntax
• Assuming sorting is required before risk check

5. You have three features with these risk scores:

Feature X: 12
Feature Y: 20
Feature Z: 20

Due to limited time, you can only test two features. Which is the best risk-based testing approach?

hard

A. Test Feature X and Feature Y because X has the lowest risk

B. Test all three features equally

C. Test Feature X and Feature Z because Z is newer

D. Test Feature Y and Feature Z because they have the highest risk

Solution

1. Step 1: Identify features with highest risk

   Feature Y and Feature Z both have the highest risk score of 20.

2. Step 2: Prioritize testing based on risk and time limit

   Since only two can be tested, choose the two with highest risk to maximize risk coverage.

3. Final Answer:

   Test Feature Y and Feature Z because they have the highest risk -> Option D

4. Quick Check:

   Test highest risk features first [OK]

Hint: Pick features with highest risk scores first [OK]

Common Mistakes:

• Testing lower risk features first
• Ignoring risk scores when time is limited
• Trying to test all features equally despite limits