Defining success criteria for agents in Agentic AI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What best defines a success criterion for an AI agent?

Imagine you have a robot that cleans your house. What is the best way to describe its success criterion?

A The robot makes a sound when it starts cleaning.

B The robot moves around without bumping into walls.

C The robot cleans every room in the house within 30 minutes.

D The robot has a battery that lasts 2 hours.

💡 Hint

Think about what shows the robot did its main job well.

A success criterion measures if the agent achieved its main goal. Cleaning every room within 30 minutes shows success. Moving without bumping or battery life are important but not the main goal. Making a sound is just a feature.

❓ Metrics

intermediate

Which metric best measures success for a delivery drone agent?

A delivery drone must drop packages at correct locations on time. Which metric best shows if it succeeded?

A Percentage of packages delivered to the correct location within the scheduled time.

B Average battery usage during flights.

C Number of flights completed per day.

D Total distance flown by the drone.

💡 Hint

Focus on the main goal: delivering packages correctly and on time.

The success metric must reflect the agent's main task. Delivering packages correctly and on time is the goal, so percentage of on-time correct deliveries is best. Battery usage, flights per day, and distance flown are secondary factors.

❓ Model Choice

advanced

Choosing a success criterion for a reinforcement learning agent

You train a reinforcement learning agent to play a game. Which success criterion is best to decide if training worked?

A The agent's average score over 100 games exceeds a set threshold.

B The agent's training loss decreases steadily during training.

C The agent's model size is smaller than 10MB.

D The agent's action distribution matches the training data.

💡 Hint

Think about what shows the agent plays the game well, not just training behavior.

Success means the agent performs well in the real task. Average score over many games shows actual performance. Training loss and action distribution relate to learning but don't guarantee good play. Model size is unrelated to success.

🔧 Debug

advanced

Why does this success criterion fail to measure agent success?

An agent's success criterion is: 'Agent completes task if total steps taken is less than 50'. The agent finishes tasks but often takes 60 steps. Why is this criterion problematic?

A It requires the agent to take more than 50 steps to succeed.

B It measures the agent's speed, which is always irrelevant.

C It uses total steps, which is not a measurable quantity.

D It ignores whether the task was actually completed correctly.

💡 Hint

Think about what the criterion checks versus what the agent actually does.

The criterion only checks if the agent is fast (less than 50 steps), but does not check if the task was done correctly. The agent may finish correctly but slowly, so the criterion misses success.

❓ Hyperparameter

expert

How does changing the success threshold affect agent training?

An agent's success criterion is reaching a score above a threshold. What happens if you set this threshold too high during training?

A The agent will learn faster because the goal is more challenging.

B The agent may never reach success, causing training to stall or fail.

C The agent ignores the threshold and trains normally.

D The agent's model size will automatically increase.

💡 Hint

Consider what happens if the goal is impossible or too hard to reach.

If the success threshold is too high, the agent may never achieve success, so training signals become weak or absent. This can cause training to stall or fail. Harder goals don't always speed learning. Model size is unrelated.

Practice

1. Why is it important to define success criteria for an AI agent?

easy

A. It reduces the size of the agent's code.

B. It helps the agent understand what goal to achieve.

C. It makes the agent run faster.

D. It allows the agent to ignore errors.

Solution

1. Step 1: Understand the role of success criteria

   Success criteria tell the agent what outcome is desired or considered good.

2. Step 2: Connect success criteria to agent behavior

   Without clear goals, the agent cannot know what to aim for or when it has succeeded.

3. Final Answer:

   It helps the agent understand what goal to achieve. -> Option B

4. Quick Check:

   Success criteria = clear goals [OK]

Hint: Success criteria define the agent's goal clearly [OK]

Common Mistakes:

• Thinking success criteria speed up the agent
• Confusing success criteria with code size
• Believing success criteria ignore errors

2. Which of the following is the correct way to express a success criterion for an agent in code?

easy

A. success == accuracy > 0.9

B. success = accuracy = 0.9

C. success = accuracy > 0.9

D. success => accuracy > 0.9

Solution

1. Step 1: Identify correct comparison syntax

   In Python, to assign a boolean result, use a single = with a comparison expression on the right.

2. Step 2: Check each option's syntax

   success = accuracy > 0.9 uses correct assignment and comparison. success = accuracy = 0.9 uses = instead of == for comparison. success == accuracy > 0.9 uses == incorrectly for assignment. success => accuracy > 0.9 uses => which is invalid in Python.

3. Final Answer:

   success = accuracy > 0.9 -> Option C

4. Quick Check:

   Assignment with comparison uses = and > [OK]

Hint: Use '=' for assignment, '>' for comparison [OK]

Common Mistakes:

• Using '==' instead of '=' for assignment
• Using '=' instead of '==' for comparison
• Using invalid operators like '=>'

3. Given the code below, what will be the value of success?

accuracy = 0.85
threshold = 0.8
success = accuracy >= threshold

medium

A. True

B. Error

C. 0.85

D. False

Solution

1. Step 1: Compare accuracy and threshold values

   Accuracy is 0.85, threshold is 0.8, so 0.85 >= 0.8 is True.

2. Step 2: Assign comparison result to success

   The boolean True is assigned to success.

3. Final Answer:

   True -> Option A

4. Quick Check:

   0.85 >= 0.8 = True [OK]

Hint: Check if accuracy meets or exceeds threshold [OK]

Common Mistakes:

• Confusing value 0.85 with boolean True
• Thinking comparison returns a number
• Expecting an error from valid comparison

4. The following code is intended to check if an agent's success metric is above 90%, but it has a bug. What is the bug?

success_metric = 0.92
if success_metric = 0.9:
    print('Agent succeeded')

medium

A. Missing colon ':' after if statement

B. Print statement syntax error

C. Incorrect variable name 'success_metric'

D. Using '=' instead of '==' in the if condition

Solution

1. Step 1: Identify the if statement syntax

   In Python, '=' is for assignment, '==' is for comparison in conditions.

2. Step 2: Locate the bug in the if condition

   The code uses '=' instead of '==' which causes a syntax error.

3. Final Answer:

   Using '=' instead of '==' in the if condition -> Option D

4. Quick Check:

   Use '==' for comparison in if [OK]

Hint: Use '==' to compare values in if statements [OK]

Common Mistakes:

• Confusing '=' with '==' in conditions
• Ignoring syntax errors from wrong operators
• Assuming missing colon is the error

5. You want to define success criteria for an agent that completes tasks with at least 95% accuracy and finishes within 10 seconds. Which of the following is the best way to define this success criteria in code?

hard

A. success = (accuracy >= 0.95) and (time_taken <= 10)

B. success = accuracy > 0.95 or time_taken < 10

C. success = accuracy == 0.95 and time_taken == 10

D. success = accuracy >= 0.95 and time_taken > 10

Solution

1. Step 1: Understand the criteria requirements

   The agent must have accuracy at least 95% and finish within 10 seconds.

2. Step 2: Translate criteria into logical conditions

   Use '>=' for accuracy and '<=' for time, combined with 'and' to require both.

3. Step 3: Evaluate each option

   success = (accuracy >= 0.95) and (time_taken <= 10) correctly uses 'and' and proper comparisons. success = accuracy > 0.95 or time_taken < 10 uses 'or' which allows passing if only one condition is met. success = accuracy == 0.95 and time_taken == 10 uses '==' which is too strict. success = accuracy >= 0.95 and time_taken > 10 allows time_taken > 10 which breaks the time limit.

4. Final Answer:

   success = (accuracy >= 0.95) and (time_taken <= 10) -> Option A

5. Quick Check:

   Both accuracy and time must meet thresholds [OK]

Hint: Use 'and' to combine all success conditions [OK]

Common Mistakes:

• Using 'or' instead of 'and' to combine conditions
• Using '==' instead of '>=' or '<='
• Allowing time greater than limit