Why complex tasks need planning in Agentic AI - Why Metrics Matter

Metrics & Evaluation - Why complex tasks need planning

Which metric matters for this concept and WHY

For complex tasks that require planning, task success rate and efficiency metrics matter most. Task success rate shows if the plan leads to completing the task correctly. Efficiency metrics, like time or steps taken, show if the plan is practical and not wasteful. These metrics help us know if the AI plans well enough to handle complexity.

Confusion matrix or equivalent visualization (ASCII)

Task Outcome Confusion Matrix:

                | Planned Success | Planned Failure |
--------------------------------------------------
Actual Success |       TP=80     |      FN=20      |
Actual Failure |       FP=10     |      TN=90      |

Total tasks = 200

- TP: Tasks planned and succeeded
- FP: Tasks planned but failed
- FN: Tasks not planned but succeeded
- TN: Tasks not planned and failed

Precision = TP / (TP + FP) = 80 / (80 + 10) = 0.89
Recall = TP / (TP + FN) = 80 / (80 + 20) = 0.80
F1 Score = 2 * (0.89 * 0.80) / (0.89 + 0.80) = 0.84
    

Precision vs Recall tradeoff with concrete examples

In planning complex tasks, precision means the plan leads to success most of the time. Recall means the plan covers most tasks that need planning.

Example: A robot plans to clean rooms. High precision means when it plans a cleaning, it usually cleans well. High recall means it plans cleaning for almost all dirty rooms.

If precision is high but recall is low, the robot cleans well but misses many dirty rooms. If recall is high but precision is low, it tries to clean many rooms but often fails.

Good planning balances both to cover tasks well and succeed often.

What "good" vs "bad" metric values look like for this use case

Good planning metrics:

• Task success rate above 85%
• Precision and recall both above 80%
• Low number of unnecessary steps (high efficiency)

Bad planning metrics:

• Task success rate below 60%
• Precision or recall below 50%
• Plans that take too long or waste resources

Metrics pitfalls

• Accuracy paradox: High overall success can hide poor planning on complex subtasks.
• Data leakage: If the AI sees future task info during planning, metrics look better but are unrealistic.
• Overfitting: Planning that works only on training tasks but fails on new ones.
• Ignoring efficiency: A plan that always succeeds but takes too long is not practical.

Self-check question

Your AI planner has 98% task success rate but only 12% recall on complex subtasks. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means it misses most complex subtasks needing planning. Even with high overall success, many important tasks are ignored, which can cause failures in real use.

Key Result

For complex tasks, balanced precision and recall with high task success rate ensure effective and efficient planning.

Practice

1. Why is planning important for complex tasks in AI systems?

easy

A. It makes the task more confusing.

B. It breaks the task into smaller, manageable steps.

C. It slows down the process.

D. It removes the need for data.

Solution

1. Step 1: Understand the role of planning

   Planning helps by dividing a big task into smaller parts that are easier to handle.

2. Step 2: Recognize the benefits for AI systems

   This division allows AI to work smarter and faster by focusing on one step at a time.

3. Final Answer:

   It breaks the task into smaller, manageable steps. -> Option B

4. Quick Check:

   Planning = breaking tasks down [OK]

Hint: Planning means splitting big tasks into small steps [OK]

Common Mistakes:

• Thinking planning makes tasks slower
• Believing planning removes data needs
• Assuming planning confuses AI

2. Which of the following is the correct way to represent a plan for a complex task in Python?

easy

A. steps = ['collect data', 'clean data', 'train model', 'evaluate']

B. steps = collect data, clean data, train model, evaluate

C. steps = {collect data; clean data; train model; evaluate}

D. steps = (collect data clean data train model evaluate)

Solution

1. Step 1: Identify correct Python list syntax

   Python lists use square brackets [] with items separated by commas.

2. Step 2: Check each option's syntax

   steps = ['collect data', 'clean data', 'train model', 'evaluate'] uses correct list syntax with strings in quotes and commas.

3. Final Answer:

   steps = ['collect data', 'clean data', 'train model', 'evaluate'] -> Option A

4. Quick Check:

   Python lists use [] and commas [OK]

Hint: Lists use [] with commas separating items [OK]

Common Mistakes:

• Missing quotes around strings
• Using commas outside brackets
• Using curly braces or parentheses incorrectly

3. Consider this Python code representing a simple plan execution:

plan = ['step1', 'step2', 'step3']
for i, step in enumerate(plan):
    print(f"Executing {step} number {i+1}")

What will be the output?

medium

A. Executing step1 number 1
Executing step2 number 2
Executing step3 number 3

B. Error: enumerate not defined

C. step1
step2
step3

D. Executing step1 number 0
Executing step2 number 1
Executing step3 number 2

Solution

1. Step 1: Understand enumerate behavior

   enumerate gives index starting at 0 and the item; i+1 shifts index to start at 1.

2. Step 2: Trace the loop output

   For each step, it prints "Executing {step} number {i+1}", so numbers start at 1.

3. Final Answer:

   Executing step1 number 1 Executing step2 number 2 Executing step3 number 3 -> Option A

4. Quick Check:

   enumerate index + 1 = printed number [OK]

Hint: enumerate index starts at 0; add 1 for counting [OK]

Common Mistakes:

• Forgetting to add 1 to index
• Confusing output format
• Assuming enumerate is undefined

4. The following code is intended to print each step of a plan with its number, but it causes an error:

plan = ['collect', 'process', 'train']
for step in plan:
    print(f"Step {i}: {step}")

What is the error and how to fix it?

medium

A. List 'plan' is empty; add items.

B. Syntax error in print statement; fix quotes.

C. Indentation error; fix loop indentation.

D. Variable 'i' is not defined; add enumerate to loop.

Solution

1. Step 1: Identify the error cause

   The variable 'i' is used but never defined in the loop.

2. Step 2: Fix by adding enumerate

   Use 'for i, step in enumerate(plan):' to define 'i' as index.

3. Final Answer:

   Variable 'i' is not defined; add enumerate to loop. -> Option D

4. Quick Check:

   Use enumerate to get index [OK]

Hint: Use enumerate to get index in loops [OK]

Common Mistakes:

• Ignoring undefined variable errors
• Trying to fix quotes instead of variable
• Assuming list is empty

5. You want an AI agent to plan a complex task: "Prepare a report". Which planning approach best helps the agent work efficiently?

hard

A. Only gather data and submit without analysis or writing.

B. Start writing the report immediately without any plan.

C. Break the task into steps: gather data, analyze, write, review, submit.

D. Ask the user to do all steps manually.

Solution

1. Step 1: Understand task complexity

   Preparing a report involves multiple stages that need order and focus.

2. Step 2: Choose a planning approach

   Breaking the task into clear steps helps the AI manage and complete each part efficiently.

3. Final Answer:

   Break the task into steps: gather data, analyze, write, review, submit. -> Option C

4. Quick Check:

   Planning = stepwise task breakdown [OK]

Hint: Divide complex tasks into clear steps [OK]

Common Mistakes:

• Skipping planning and starting immediately
• Ignoring important steps like analysis
• Delegating all work to user