Regression testing for agent changes in Agentic AI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why is regression testing important after updating an AI agent?

Imagine you have an AI agent that helps users book flights. You update its code to improve speed. Why should you run regression tests after this update?

A To add new features to the agent

B To check if the update accidentally broke existing features

C To improve the agent's speed further

D To delete old data from the system

💡 Hint

Think about what might happen to old features when you change something new.

Regression testing ensures that new changes do not break existing functionalities. It helps keep the agent reliable.

❓ Predict Output

intermediate

Output of regression test result summary

Given this Python code simulating regression test results for an AI agent update, what will be printed?

Agentic AI

test_results = {'login': True, 'search': True, 'booking': False, 'payment': True}
failed_tests = [k for k, v in test_results.items() if not v]
print(f"Failed tests: {failed_tests}")

A Failed tests: ['login', 'search']

B Failed tests: ['payment']

C Failed tests: []

D Failed tests: ['booking']

💡 Hint

Look for tests with value False in the dictionary.

The code collects keys where the test result is False. Only 'booking' test failed.

❓ Model Choice

advanced

Choosing a model for regression testing of agent behavior

You want to predict if an AI agent update will cause failures in certain tasks based on past update data. Which model is best suited for this binary classification problem?

A Linear Regression

B K-Means Clustering

C Logistic Regression

D Principal Component Analysis (PCA)

💡 Hint

Think about models that predict categories like pass/fail.

Logistic Regression is used for binary classification problems like predicting pass/fail outcomes.

❓ Metrics

advanced

Interpreting regression test classification metrics

An AI agent regression test classifier has these results: 90 true positives, 10 false positives, 5 false negatives, and 95 true negatives. What is the precision of the classifier?

A 90 / (90 + 10) = 0.9

B 90 / (90 + 5) = 0.947

C 95 / (95 + 10) = 0.905

D 90 / (90 + 95) = 0.486

💡 Hint

Precision = True Positives / (True Positives + False Positives)

Precision measures how many predicted positives are actually correct.

🔧 Debug

expert

Debugging a failing regression test script for agent changes

Consider this Python code snippet that runs regression tests on an AI agent's functions. It should print 'All tests passed' if all tests return True, else print failed test names. What error or output will this code produce?

Agentic AI

def test_login():
    return True

def test_search():
    return False

tests = {'login': test_login, 'search': test_search}

failed = []
for name, func in tests.items():
    if func() == False:
        failed.append(name)

if failed:
    print('Failed tests:', failed)
else:
    print('All tests passed')

A Failed tests: ['search']

B All tests passed

C TypeError because func() is not callable

D SyntaxError due to missing colon

💡 Hint

Check which test returns False and how the code collects failures.

The 'search' test returns False, so it is added to failed list and printed.

Practice

1. What is the main purpose of regression testing for agent changes?

easy

A. To check if new changes break old agent behavior

B. To improve the agent's speed

C. To add new features to the agent

D. To change the agent's user interface

Solution

1. Step 1: Understand regression testing goal

   Regression testing is done to ensure that recent changes do not break existing functionality.

2. Step 2: Match purpose with options

   To check if new changes break old agent behavior clearly states checking if new changes break old behavior, which matches the goal.

3. Final Answer:

   To check if new changes break old agent behavior -> Option A

4. Quick Check:

   Regression testing = check old behavior intact [OK]

Hint: Regression testing checks old features still work after changes [OK]

Common Mistakes:

• Thinking regression testing adds new features
• Confusing regression testing with performance testing
• Assuming regression testing changes UI

2. Which of the following is the correct way to define a test case for regression testing an agent in Python?

easy

A. def test_agent(): assert agent.run(input) == expected_output

B. test agent run input equals expected output

C. def test_agent: return agent.run(input) == expected_output

D. function test_agent() { return agent.run(input) == expected_output; }

Solution

1. Step 1: Identify correct Python function syntax

   Python functions start with 'def', have parentheses, and a colon.

2. Step 2: Check assertion usage

   def test_agent(): assert agent.run(input) == expected_output uses 'assert' correctly to compare output, matching Python test style.

3. Final Answer:

   def test_agent(): assert agent.run(input) == expected_output -> Option A

4. Quick Check:

   Python test function with assert = def test_agent(): assert agent.run(input) == expected_output [OK]

Hint: Python test functions start with def and use assert [OK]

Common Mistakes:

• Missing parentheses or colon in function definition
• Using non-Python syntax
• Not using assert for test checks

3. Given the code below, what will be the output of the regression test?

class Agent:
    def run(self, x):
        return x * 2

def test_agent():
    agent = Agent()
    result = agent.run(3)
    assert result == 6
    print('Test passed')

test_agent()

medium

A. SyntaxError

B. Test passed

C. AssertionError

D. No output

Solution

1. Step 1: Understand agent run method

   The method multiplies input by 2, so run(3) returns 6.

2. Step 2: Check assertion and print

   The assertion checks if result == 6, which is true, so no error occurs and 'Test passed' prints.

3. Final Answer:

   Test passed -> Option B

4. Quick Check:

   3 * 2 = 6, assertion true, prints message [OK]

Hint: Check method output matches assertion to predict test result [OK]

Common Mistakes:

• Assuming assertion fails without checking output
• Confusing syntax errors with logic errors
• Ignoring print statement after assertion

4. Identify the error in the following regression test code and select the fix:

def test_agent():
    agent = Agent()
    result = agent.run(5)
    if result = 10:
        print('Test passed')
    else:
        print('Test failed')

medium

A. Replace print with return statements

B. Add parentheses around the if condition

C. Change '=' to '==' in the if condition

D. Remove else block

Solution

1. Step 1: Identify syntax error in if condition

   The single '=' is an assignment, not a comparison, causing a syntax error.

2. Step 2: Correct the comparison operator

   Replace '=' with '==' to compare values properly in the if statement.

3. Final Answer:

   Change '=' to '==' in the if condition -> Option C

4. Quick Check:

   Use '==' for comparison in if statements [OK]

Hint: Use '==' to compare, '=' assigns values [OK]

Common Mistakes:

• Using '=' instead of '==' in conditions
• Adding unnecessary parentheses in Python if
• Thinking print must be replaced with return

5. You updated your agent's decision logic. How should you design regression tests to ensure old behaviors remain correct while testing new features?

hard

A. Test randomly without expected outputs to save time

B. Only test new features since old ones worked before

C. Remove old tests to avoid conflicts with new logic

D. Create test cases for both old expected outputs and new expected outputs

Solution

1. Step 1: Understand regression test purpose

   Regression tests verify that old behaviors still work after changes.

2. Step 2: Design tests covering old and new behaviors

   Include test cases for old expected outputs and new expected outputs to check both.

3. Final Answer:

   Create test cases for both old expected outputs and new expected outputs -> Option D

4. Quick Check:

   Test old and new outputs to ensure full correctness [OK]

Hint: Test old and new cases to catch breaks early [OK]

Common Mistakes:

• Ignoring old tests after updates
• Deleting old tests to simplify
• Skipping expected outputs in tests