Regression testing in Software Engineering - Mini Project: Build & Apply

Understanding Regression Testing

📖 Scenario: You are a QA engineer managing a regression test suite for a web application. After a code change, you need to identify affected modules, select the right tests, run them, and analyze the results to decide if the code is safe to deploy.

🎯 Goal: Build a Python model that simulates regression testing: define modules and their tests, identify affected modules from a code change, run the selected tests, and determine deployment readiness.

📋 What You'll Learn

Create a dictionary mapping modules to their test cases

Identify affected modules based on a code change

Select and run regression tests for affected modules

Determine deployment readiness based on test results

💡 Why This Matters

🌍 Real World

Regression testing is fundamental to CI/CD pipelines. Every major software company runs regression suites automatically on every commit to catch regressions before they reach production.

💼 Career

QA engineers, SDETs, and developers are expected to understand regression testing strategies, test selection, and how to analyze failures in technical interviews.

1

Define modules and their test cases

Create a dictionary called module_tests with three entries: 'auth' mapped to ['test_login', 'test_logout', 'test_token'], 'cart' mapped to ['test_add_item', 'test_remove_item'], and 'payment' mapped to ['test_checkout', 'test_refund', 'test_receipt'].

Software Engineering

# Create the module_tests dictionary
# Your code here

Hint

Each module key maps to a list of test function names that verify that module's behavior.

2

Identify affected modules

Create a list called changed_modules containing 'auth'. Create a dictionary called dependencies with 'auth' mapped to ['cart'], 'cart' mapped to ['payment'], and 'payment' mapped to an empty list []. Create an empty list called affected_modules. Use a for loop with variable module over changed_modules to append each module and its dependencies from dependencies[module] to affected_modules.

Software Engineering

module_tests = {
    'auth': ['test_login', 'test_logout', 'test_token'],
    'cart': ['test_add_item', 'test_remove_item'],
    'payment': ['test_checkout', 'test_refund', 'test_receipt']
}
# Identify affected modules from change in auth
# Your code here

Hint

When auth changes, both auth and its dependent module cart need regression testing.

3

Select and run regression tests

Create an empty list called selected_tests. Use a for loop with variable module over affected_modules to extend selected_tests with module_tests[module]. Create a dictionary called test_results. Use a for loop with variable test over selected_tests to set each test's result to 'pass' in test_results. Then override test_results['test_token'] to 'fail' to simulate a regression.

Software Engineering

module_tests = {
    'auth': ['test_login', 'test_logout', 'test_token'],
    'cart': ['test_add_item', 'test_remove_item'],
    'payment': ['test_checkout', 'test_refund', 'test_receipt']
}
changed_modules = ['auth']
dependencies = {
    'auth': ['cart'],
    'cart': ['payment'],
    'payment': []
}
affected_modules = []
for module in changed_modules:
    affected_modules.append(module)
    affected_modules.extend(dependencies[module])
# Select tests, run them, and simulate one failure
# Your code here

Hint

Collect all tests for affected modules, run them all as pass, then override one to simulate a failure.

4

Determine deployment readiness

Create a list called failures using a list comprehension that collects each test from test_results where test_results[test] equals 'fail'. Create a variable deploy_ready set to len(failures) == 0.

Software Engineering

module_tests = {
    'auth': ['test_login', 'test_logout', 'test_token'],
    'cart': ['test_add_item', 'test_remove_item'],
    'payment': ['test_checkout', 'test_refund', 'test_receipt']
}
changed_modules = ['auth']
dependencies = {
    'auth': ['cart'],
    'cart': ['payment'],
    'payment': []
}
affected_modules = []
for module in changed_modules:
    affected_modules.append(module)
    affected_modules.extend(dependencies[module])
selected_tests = []
for module in affected_modules:
    selected_tests.extend(module_tests[module])
test_results = {}
for test in selected_tests:
    test_results[test] = 'pass'
test_results['test_token'] = 'fail'
# Determine failures and deployment readiness
# Your code here

Hint

If any test failed, the code is not ready to deploy. deploy_ready should be False when failures exist.

Practice

1. What is the main purpose of regression testing in software development?

easy

A. To check that new code changes do not break existing features

B. To test the software performance under heavy load

C. To find security vulnerabilities in the software

D. To design the user interface of the application

Solution

1. Step 1: Understand regression testing goal

   Regression testing is done to ensure that recent code changes do not negatively affect existing functionality.

2. Step 2: Match purpose with options

   To check that new code changes do not break existing features clearly states this purpose, while others focus on performance, security, or design, which are different testing types.

3. Final Answer:

   To check that new code changes do not break existing features -> Option A

4. Quick Check:

   Regression testing = check existing features still work [OK]

Hint: Regression testing = verify old features after changes [OK]

Common Mistakes:

• Confusing regression testing with performance testing
• Thinking regression testing finds new bugs only
• Mixing regression testing with security testing

2. Which of the following is the correct way to describe an automated regression test?

easy

A. A script that runs tests automatically after code changes

B. A report summarizing user feedback

C. A manual checklist to verify UI changes

D. A design document for new features

Solution

1. Step 1: Identify automation in regression testing

   Automated regression tests are scripts that run automatically to check software after changes.

2. Step 2: Compare options to automation definition

   A script that runs tests automatically after code changes matches this exactly. Others describe manual or unrelated activities.

3. Final Answer:

   A script that runs tests automatically after code changes -> Option A

4. Quick Check:

   Automated regression test = automatic script [OK]

Hint: Automation means tests run by script, not manually [OK]

Common Mistakes:

• Confusing manual checks with automated tests
• Thinking reports or documents are tests
• Ignoring the 'automatic' part of regression testing

3. Consider this Python code snippet for a regression test automation check:

def test_model_accuracy(model, test_data, expected_accuracy):
    accuracy = model.evaluate(test_data)
    return accuracy >= expected_accuracy

class DummyModel:
    def evaluate(self, data):
        return 0.85

model = DummyModel()
result = test_model_accuracy(model, None, 0.80)
print(result)

What will be the output when this code runs?

medium

A. Error: 'NoneType' object is not iterable

B. False

C. True

D. 0.85

Solution

1. Step 1: Understand the evaluate method output

   The DummyModel's evaluate method returns 0.85 regardless of input.

2. Step 2: Check the comparison in test_model_accuracy

   It compares 0.85 >= 0.80, which is True, so the function returns True.

3. Final Answer:

   True -> Option C

4. Quick Check:

   0.85 >= 0.80 = True [OK]

Hint: Check returned value and compare to expected threshold [OK]

Common Mistakes:

• Assuming None input causes error
• Confusing returned accuracy with boolean result
• Ignoring the fixed return value of evaluate

4. You have this regression test code snippet in Python:

def run_regression_tests(tests):
    for test in tests:
        if not test():
            print("Test failed")
            break
    else:
        print("All tests passed")

def test1():
    return True

def test2():
    return False

run_regression_tests([test1, test2])

What is the output of this code?

medium

A. All tests passed

B. Test failed

C. No output

D. Error: break outside loop

Solution

1. Step 1: Analyze the loop over tests

   The loop runs test1() which returns True, so continues to test2().

2. Step 2: Check test2() result and loop behavior

   test2() returns False, triggering print("Test failed") and break to exit loop.

3. Final Answer:

   Test failed -> Option B

4. Quick Check:

   Second test fails, loop breaks, prints 'Test failed' [OK]

Hint: Loop breaks on first failed test, prints failure [OK]

Common Mistakes:

• Thinking else after for runs always
• Ignoring break statement effect
• Assuming all tests run regardless of failure

5. You want to improve your regression testing for a machine learning model that predicts house prices. Which approach best ensures your tests catch errors after model updates?

hard

A. Skip testing if training loss decreases

B. Manually check a few random predictions after each update

C. Only test the model when users report errors

D. Run automated tests comparing model predictions on a fixed test set before and after updates

Solution

1. Step 1: Identify best regression testing practice for ML models

   Automated tests on a fixed test set ensure consistent checks after updates.

2. Step 2: Evaluate options for reliability

   Run automated tests comparing model predictions on a fixed test set before and after updates uses automation and fixed data, catching errors reliably. Others rely on manual or reactive methods, which are less reliable.

3. Final Answer:

   Run automated tests comparing model predictions on a fixed test set before and after updates -> Option D

4. Quick Check:

   Automated fixed tests catch errors best [OK]

Hint: Use automated tests on fixed data to catch changes [OK]

Common Mistakes:

• Relying on manual checks only
• Ignoring testing until errors reported
• Assuming lower training loss means no bugs