Regression testing in Software Engineering - Step-by-Step Execution

Concept Flow - Regression testing

Start: Code change made (bug fix, feature, refactor)

↓

Identify affected modules

↓

Select regression test suite

↓

Run selected tests against updated code

↓

All tests pass?

Yes→Deploy with confidence

No↓

Identify failing tests

↓

Is failure a real bug or intended change?

↓

Real bug: Fix code

↓

Re-run regression suite

↓

End

This flow shows how regression testing verifies that new code changes do not break existing functionality by running a suite of previously passing tests.

Execution Sample

Software Engineering

Code change: Fix login timeout bug
Identify affected: auth module, session module
Select tests: 45 auth tests + 12 session tests + 20 smoke tests
Run suite: 77 tests executed
Results: 75 pass, 2 fail
Analyze failures: session_expiry_test, token_refresh_test
Diagnosis: timeout fix broke token refresh logic
Fix: Update token refresh to use new timeout value
Re-run: 77/77 pass

This sequence shows a regression testing cycle after a bug fix, where the fix inadvertently broke two related tests.

Analysis Table

Step	Action	Condition/Check	Result/Decision	Next Step
1	Developer fixes login timeout bug	N/A	Code changed in auth module	Identify affected modules
2	Identify modules affected by change	Dependency analysis	auth + session modules affected	Select test suite
3	Select regression test suite	Based on affected modules + smoke tests	77 tests selected	Run tests
4	Run regression tests	Execute all 77 tests	75 pass, 2 fail	Analyze failures
5	Analyze failing tests	Are failures real bugs or intended?	Real bugs — token refresh broken	Fix the code
6	Fix token refresh logic	N/A	Code updated	Re-run regression suite
7	Re-run regression suite	All 77 tests	77/77 pass	End — safe to deploy

💡 Regression testing confirmed the fix works without breaking existing functionality. The code is safe to deploy.

State Tracker

Variable	Start	After Step 2	After Step 4	After Step 6	Final
code_state	Timeout bug present	Bug fixed, untested	2 regressions found	Regressions fixed	All tests pass
tests_selected	0	0	77	77	77
tests_passing	N/A	N/A	75/77	N/A	77/77
deploy_ready	No	No	No (failures)	No (untested)	Yes

Key Insights - 3 Insights

Why do we run regression tests after every code change?

Code changes can have unintended side effects on existing functionality. Regression testing catches these regressions before they reach production, as demonstrated in step 4 where the timeout fix broke token refresh.

How do you decide which tests to include in the regression suite?

Tests are selected based on the modules affected by the change plus smoke tests that cover critical paths. As shown in step 3, dependency analysis identifies that auth changes also affect the session module.

What is the difference between a real regression and an intended behavior change?

A real regression means the code change broke something unintentionally (fix the code). An intended change means the old test expectation is outdated (update the test). Step 5 shows this decision point.

Visual Quiz - 3 Questions

Look at the execution_table at step 4. What happened when the regression suite was run?

A All 77 tests passed

B 75 tests passed and 2 failed

C The test suite could not run

D Only smoke tests were executed

💡 Hint

Check the 'Result/Decision' column in step 4.

According to the variable_tracker, when does deploy_ready become Yes?

A After the initial code fix

B After selecting the test suite

C After all 77 tests pass in the final run

D After analyzing the failing tests

💡 Hint

Look at the deploy_ready row across all columns.

Why were session module tests included in the regression suite even though the fix was in the auth module?

A Session tests are always run regardless

B The session module depends on the auth module

C Session tests are faster to run

D The developer requested them manually

💡 Hint

Check step 2 where dependency analysis identifies affected modules.

Concept Snapshot

Regression testing verifies that code changes do not break existing functionality.
After every change: identify affected modules, select tests, run suite.
Failures are analyzed: real bugs get fixed, intended changes update tests.
The cycle repeats until all tests pass — then deploy is safe.
Automation makes regression testing practical for frequent releases.

Full Transcript

This visual execution shows regression testing in action. After a developer fixes a login timeout bug, the team identifies that both the auth and session modules are affected. A regression suite of 77 tests is selected and run. Two tests fail, revealing that the timeout fix broke the token refresh logic. The team fixes the code and re-runs the suite, getting all 77 tests to pass. Only then is the code considered safe to deploy. This cycle of change, test, fix, and re-test is the core of regression testing.

Practice

1. What is the main purpose of regression testing in software development?

easy

A. To check that new code changes do not break existing features

B. To test the software performance under heavy load

C. To find security vulnerabilities in the software

D. To design the user interface of the application

Solution

1. Step 1: Understand regression testing goal

   Regression testing is done to ensure that recent code changes do not negatively affect existing functionality.

2. Step 2: Match purpose with options

   To check that new code changes do not break existing features clearly states this purpose, while others focus on performance, security, or design, which are different testing types.

3. Final Answer:

   To check that new code changes do not break existing features -> Option A

4. Quick Check:

   Regression testing = check existing features still work [OK]

Hint: Regression testing = verify old features after changes [OK]

Common Mistakes:

• Confusing regression testing with performance testing
• Thinking regression testing finds new bugs only
• Mixing regression testing with security testing

2. Which of the following is the correct way to describe an automated regression test?

easy

A. A script that runs tests automatically after code changes

B. A report summarizing user feedback

C. A manual checklist to verify UI changes

D. A design document for new features

Solution

1. Step 1: Identify automation in regression testing

   Automated regression tests are scripts that run automatically to check software after changes.

2. Step 2: Compare options to automation definition

   A script that runs tests automatically after code changes matches this exactly. Others describe manual or unrelated activities.

3. Final Answer:

   A script that runs tests automatically after code changes -> Option A

4. Quick Check:

   Automated regression test = automatic script [OK]

Hint: Automation means tests run by script, not manually [OK]

Common Mistakes:

• Confusing manual checks with automated tests
• Thinking reports or documents are tests
• Ignoring the 'automatic' part of regression testing

3. Consider this Python code snippet for a regression test automation check:

def test_model_accuracy(model, test_data, expected_accuracy):
    accuracy = model.evaluate(test_data)
    return accuracy >= expected_accuracy

class DummyModel:
    def evaluate(self, data):
        return 0.85

model = DummyModel()
result = test_model_accuracy(model, None, 0.80)
print(result)

What will be the output when this code runs?

medium

A. Error: 'NoneType' object is not iterable

B. False

C. True

D. 0.85

Solution

1. Step 1: Understand the evaluate method output

   The DummyModel's evaluate method returns 0.85 regardless of input.

2. Step 2: Check the comparison in test_model_accuracy

   It compares 0.85 >= 0.80, which is True, so the function returns True.

3. Final Answer:

   True -> Option C

4. Quick Check:

   0.85 >= 0.80 = True [OK]

Hint: Check returned value and compare to expected threshold [OK]

Common Mistakes:

• Assuming None input causes error
• Confusing returned accuracy with boolean result
• Ignoring the fixed return value of evaluate

4. You have this regression test code snippet in Python:

def run_regression_tests(tests):
    for test in tests:
        if not test():
            print("Test failed")
            break
    else:
        print("All tests passed")

def test1():
    return True

def test2():
    return False

run_regression_tests([test1, test2])

What is the output of this code?

medium

A. All tests passed

B. Test failed

C. No output

D. Error: break outside loop

Solution

1. Step 1: Analyze the loop over tests

   The loop runs test1() which returns True, so continues to test2().

2. Step 2: Check test2() result and loop behavior

   test2() returns False, triggering print("Test failed") and break to exit loop.

3. Final Answer:

   Test failed -> Option B

4. Quick Check:

   Second test fails, loop breaks, prints 'Test failed' [OK]

Hint: Loop breaks on first failed test, prints failure [OK]

Common Mistakes:

• Thinking else after for runs always
• Ignoring break statement effect
• Assuming all tests run regardless of failure

5. You want to improve your regression testing for a machine learning model that predicts house prices. Which approach best ensures your tests catch errors after model updates?

hard

A. Skip testing if training loss decreases

B. Manually check a few random predictions after each update

C. Only test the model when users report errors

D. Run automated tests comparing model predictions on a fixed test set before and after updates

Solution

1. Step 1: Identify best regression testing practice for ML models

   Automated tests on a fixed test set ensure consistent checks after updates.

2. Step 2: Evaluate options for reliability

   Run automated tests comparing model predictions on a fixed test set before and after updates uses automation and fixed data, catching errors reliably. Others rely on manual or reactive methods, which are less reliable.

3. Final Answer:

   Run automated tests comparing model predictions on a fixed test set before and after updates -> Option D

4. Quick Check:

   Automated fixed tests catch errors best [OK]

Hint: Use automated tests on fixed data to catch changes [OK]

Common Mistakes:

• Relying on manual checks only
• Ignoring testing until errors reported
• Assuming lower training loss means no bugs