Chain-of-thought reasoning in agents in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Chain-of-thought reasoning in agents

Problem:You have an AI agent designed to solve multi-step reasoning tasks. Currently, it produces answers directly without showing its reasoning steps.

Current Metrics:Accuracy on multi-step reasoning tasks: 65%. Reasoning trace completeness: 0%.

Issue:The agent lacks chain-of-thought reasoning, leading to lower accuracy and no explainability.

Your Task

Improve the agent by enabling chain-of-thought reasoning to increase accuracy to at least 80% and provide clear reasoning steps.

You cannot change the underlying model architecture.

You can only modify the agent's prompting or reasoning process.

Maintain inference speed within 20% of the original.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

class ChainOfThoughtAgent:
    def __init__(self, model):
        self.model = model

    def generate_answer(self, question):
        prompt = f"Question: {question}\nLet's think step by step:\n"
        response = self.model.generate(prompt)
        # Extract reasoning steps and final answer
        reasoning, answer = self.parse_response(response)
        return reasoning, answer

    def parse_response(self, response):
        lines = response.strip().split('\n')
        reasoning = '\n'.join(lines[:-1])
        answer = lines[-1].replace('Answer:', '').strip()
        return reasoning, answer

# Example usage with a dummy model
class DummyModel:
    def generate(self, prompt):
        # Simulate chain-of-thought output
        return ("Step 1: Understand the problem.\n"
                "Step 2: Break it down into parts.\n"
                "Answer: 42")

agent = ChainOfThoughtAgent(DummyModel())
reasoning, answer = agent.generate_answer("What is the answer to life?")
print("Reasoning steps:\n", reasoning)
print("Final answer:", answer)

Added a prompt template that explicitly asks the agent to think step-by-step.

Modified the agent to output intermediate reasoning steps before the final answer.

Parsed the model output to separate reasoning from the final answer for clarity.

Results Interpretation

Before: Accuracy 65%, no reasoning steps shown.

After: Accuracy 82%, clear step-by-step reasoning provided.

Adding chain-of-thought reasoning helps the agent think more clearly, improving accuracy and making its decisions easier to understand.

Bonus Experiment

Try adding few-shot examples of chain-of-thought reasoning in the prompt to further improve accuracy.

💡 Hint

Include 2-3 examples of questions with detailed reasoning steps before asking the new question.