Multi-step reasoning in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Multi-step reasoning

Which metric matters for Multi-step reasoning and WHY

Multi-step reasoning tasks require the model to correctly follow a chain of logic steps to reach the right answer. Because of this, accuracy is important to measure how often the model gets the full reasoning correct. However, since errors can happen at any step, precision and recall on intermediate reasoning steps or sub-tasks can also be useful to understand where mistakes occur. Overall, accuracy tells us if the model solves the whole problem correctly, while precision and recall help diagnose partial errors.

Confusion matrix for Multi-step reasoning

      | Predicted Correct | Predicted Incorrect |
      |-------------------|---------------------|
      | True Positive (TP) | False Negative (FN) |
      | False Positive (FP)| True Negative (TN)  |

      TP: Model correctly completes all reasoning steps.
      FN: Model predicted incorrect when reasoning is actually correct.
      FP and TN are less common but can represent partial step correctness in some setups.

      Example counts:
      TP = 80, FN = 20, FP = 5, TN = 95
      Total samples = 200
    

Precision vs Recall tradeoff in Multi-step reasoning

Imagine a model that tries to solve math word problems step-by-step.

• High precision means when the model says a step is correct, it usually is. This avoids false positives but might miss some correct steps.
• High recall means the model finds most of the correct steps, but might also include some wrong ones.

For multi-step reasoning, high recall is important to catch all correct steps, but high precision ensures the reasoning is reliable. Balancing both with the F1 score helps measure overall step correctness.

What "good" vs "bad" metric values look like for Multi-step reasoning

Good: Accuracy above 85% means the model solves most problems fully correct. Precision and recall above 80% on reasoning steps show reliable and complete logic.

Bad: Accuracy below 50% means the model often fails to complete reasoning. Precision or recall below 50% on steps means many errors or missed logic, making the model unreliable.

Common pitfalls in evaluating Multi-step reasoning

• Accuracy paradox: High accuracy can be misleading if the dataset has many easy problems and few hard ones.
• Data leakage: If the model sees answers during training, metrics will be unrealistically high.
• Overfitting: Model performs well on training but poorly on new problems, showing low generalization.
• Ignoring intermediate steps: Only checking final answer misses errors in reasoning steps.

Self-check question

Your multi-step reasoning model has 98% accuracy but only 12% recall on intermediate reasoning steps. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy means it often gets the final answer right, but the very low recall on steps means it misses most correct intermediate steps. This suggests the model might guess or shortcut reasoning, which can fail on harder problems or reduce trust in explanations.

Key Result

Accuracy measures full reasoning correctness; precision and recall on steps diagnose partial errors for multi-step reasoning.

Practice

1.

What does multi-step reasoning help an AI model do?

easy

A. Solve problems by breaking them into smaller steps

B. Answer questions with a single fact only

C. Ignore the order of information

D. Randomly guess answers without logic

Solution

1. Step 1: Understand the meaning of multi-step reasoning

   Multi-step reasoning means solving problems step-by-step, using several facts or actions in order.

2. Step 2: Match the meaning to the options

   Solve problems by breaking them into smaller steps says breaking problems into smaller steps, which matches the meaning exactly.

3. Final Answer:

   Solve problems by breaking them into smaller steps -> Option A

4. Quick Check:

   Multi-step reasoning = step-by-step solving [OK]

Hint: Think: Does the option show step-by-step solving? [OK]

Common Mistakes:

• Choosing options that ignore order
• Picking answers about guessing
• Confusing single fact with multiple steps

2.

Which of the following is the correct syntax to start a multi-step reasoning process in Python?

def reasoning_process():
    step1 = 'Gather data'
    step2 = 'Analyze data'
    # What comes next?

easy

A. print(step1, step2)

B. step3 = 'Make decision'

C. return step1 + step2

D. step1 = step2

Solution

1. Step 1: Understand the code context

   The function defines step1 and step2 as strings describing reasoning steps.

2. Step 2: Identify the next step in multi-step reasoning

   step3 = 'Make decision' adds a new step3, continuing the reasoning process logically.

3. Final Answer:

   step3 = 'Make decision' -> Option B

4. Quick Check:

   Next step in reasoning = add new step variable [OK]

Hint: Look for option that adds a new step logically [OK]

Common Mistakes:

• Choosing return too early
• Using print instead of continuing steps
• Overwriting previous steps

3.

What will be the output of this Python code that simulates multi-step reasoning?

def multi_step():
    step1 = 5
    step2 = step1 * 2
    step3 = step2 - 3
    return step3

print(multi_step())

medium

A. 5

B. 10

C. 7

D. None

Solution

1. Step 1: Calculate step2 from step1

   step1 = 5, so step2 = 5 * 2 = 10.

2. Step 2: Calculate step3 from step2

   step3 = 10 - 3 = 7, which is returned and printed.

3. Final Answer:

   7 -> Option C

4. Quick Check:

   5*2-3 = 7 [OK]

Hint: Calculate each step in order, then return last value [OK]

Common Mistakes:

• Returning step2 instead of step3
• Miscomputing multiplication or subtraction
• Confusing return with print output

4.

Find the error in this multi-step reasoning function and choose the fix:

def reasoning():
    step1 = 10
    step2 = step1 / 0
    step3 = step2 + 5
    return step3

medium

A. Add try-except block to handle error

B. Change division by zero to division by 1

C. Return step1 instead of step3

D. Remove step3 calculation

Solution

1. Step 1: Identify the error in the code

   Division by zero in step2 causes a runtime error (ZeroDivisionError).

2. Step 2: Choose the best fix to handle the error

   Adding a try-except block safely handles the error without stopping the program.

3. Final Answer:

   Add try-except block to handle error -> Option A

4. Quick Check:

   Division by zero needs error handling [OK]

Hint: Look for division by zero and handle with try-except [OK]

Common Mistakes:

• Ignoring the division by zero error
• Removing steps instead of fixing error
• Returning wrong variable

5.

You want to build an AI that answers questions by reasoning through three steps: understanding the question, searching facts, and giving an answer. Which approach best models this multi-step reasoning?

hard

A. Use a single neural network layer to predict answers directly

B. Randomly select an answer from a database without processing

C. Train a model only on final answers without intermediate steps

D. Chain three separate models: one for understanding, one for searching, one for answering

Solution

1. Step 1: Understand the multi-step reasoning requirement

   The AI must perform three ordered steps: understand, search, answer.

2. Step 2: Match the approach that models these steps clearly

   Chain three separate models: one for understanding, one for searching, one for answering chains three models, each handling one step, matching the multi-step reasoning process.

3. Final Answer:

   Chain three separate models: one for understanding, one for searching, one for answering -> Option D

4. Quick Check:

   Multi-step reasoning = chain models for each step [OK]

Hint: Choose option that splits tasks into ordered steps [OK]

Common Mistakes:

• Using one model for all steps ignoring order
• Random guessing without reasoning
• Skipping intermediate reasoning steps