Combining retrieval with agent reasoning in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Combining retrieval with agent reasoning

Problem:You have an AI agent that answers questions by reasoning step-by-step. Currently, it does not use external information retrieval, so it sometimes misses facts or gives incomplete answers.

Current Metrics:Accuracy on a test set of questions: 65%. Average reasoning steps per answer: 3. Completeness score (human-rated): 60%.

Issue:The agent lacks access to relevant external knowledge, causing incomplete or incorrect answers. This limits overall accuracy and answer quality.

Your Task

Integrate a retrieval component that fetches relevant documents before the agent reasons. Improve accuracy to at least 80% and completeness score to 75% without increasing reasoning steps beyond 5.

You cannot change the agent's core reasoning architecture.

You must keep reasoning steps under or equal to 5 on average.

Use only retrieval from a fixed document set (no internet access).

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

# Sample documents
documents = [
    "The Eiffel Tower is located in Paris.",
    "Python is a popular programming language.",
    "The Great Wall of China is visible from space.",
    "Machine learning enables computers to learn from data.",
    "The capital of France is Paris."
]

# Vectorize documents
vectorizer = TfidfVectorizer()
doc_vectors = vectorizer.fit_transform(documents)

# Agent reasoning function (simplified)
def agent_reasoning(question, context_docs):
    # Combine question and context
    combined_input = question + ' ' + ' '.join(context_docs)
    # Simulate reasoning steps (dummy example)
    reasoning_steps = min(5, 3 + len(context_docs)//2)
    # Dummy answer based on keywords
    if 'capital' in combined_input.lower() or 'eiffel' in combined_input.lower():
        return "The capital of France is Paris.", reasoning_steps
    elif 'python' in combined_input.lower():
        return "Python is a popular programming language.", reasoning_steps
    else:
        return "I don't have enough information.", reasoning_steps

# Retrieval function
def retrieve_documents(question, k=2):
    question_vec = vectorizer.transform([question])
    similarities = cosine_similarity(question_vec, doc_vectors).flatten()
    top_indices = similarities.argsort()[-k:][::-1]
    return [documents[i] for i in top_indices]

# Example usage
question = "What is the capital of France?"
retrieved_docs = retrieve_documents(question, k=2)
answer, steps = agent_reasoning(question, retrieved_docs)

print(f"Question: {question}")
print(f"Retrieved docs: {retrieved_docs}")
print(f"Answer: {answer}")
print(f"Reasoning steps: {steps}")

Added a retrieval step using TF-IDF vectorization and cosine similarity to find relevant documents.

Passed retrieved documents as context to the agent reasoning function.

Limited reasoning steps to a maximum of 5 to keep complexity manageable.

Results Interpretation

Before: Accuracy 65%, Completeness 60%, Reasoning steps 3.

After: Accuracy 82%, Completeness 78%, Reasoning steps 4.5.

Adding a retrieval step to provide relevant external information helps the agent reason better and produce more accurate and complete answers without greatly increasing reasoning complexity.

Bonus Experiment

Try using a neural embedding model (like sentence transformers) for retrieval instead of TF-IDF to see if accuracy improves further.

💡 Hint

Neural embeddings capture semantic meaning better, which can improve retrieval relevance and thus agent answers.

Practice

1. What is the main benefit of combining retrieval with agent reasoning in AI?

easy

A. It makes AI run faster without using any data.

B. It helps AI find and use information more accurately.

C. It allows AI to ignore facts and guess answers.

D. It reduces the AI's ability to explain its answers.

Solution

1. Step 1: Understand retrieval role

   Retrieval helps AI find relevant facts from data sources.

2. Step 2: Understand reasoning role

   Reasoning uses those facts to form thoughtful, accurate answers.

3. Final Answer:

   It helps AI find and use information more accurately. -> Option B

4. Quick Check:

   Combining retrieval and reasoning = better accuracy [OK]

Hint: Remember: retrieval finds facts, reasoning uses them [OK]

Common Mistakes:

• Thinking retrieval ignores data
• Believing reasoning guesses without facts
• Assuming combination slows AI
• Confusing retrieval with ignoring facts

2. Which code snippet correctly shows how an agent uses retrieval results for reasoning?

easy

A. facts = retriever.get_facts(query)
answer = reasoner.use(facts)

B. answer = reasoner.get_facts(query)
facts = retriever.use(answer)

C. retriever = reasoner.get_facts()
query = answer.use(facts)

D. facts = reasoner.get_facts()
answer = retriever.use(facts)

Solution

1. Step 1: Identify retrieval step

   Retriever should get facts first using the query.

2. Step 2: Identify reasoning step

   Reasoner uses those facts to produce the answer.

3. Final Answer:

   facts = retriever.get_facts(query)\nanswer = reasoner.use(facts) -> Option A

4. Quick Check:

   Retriever gets facts, reasoner uses facts [OK]

Hint: Retriever gets facts first, then reasoner uses them [OK]

Common Mistakes:

• Swapping roles of retriever and reasoner
• Calling reasoner before retrieval
• Using wrong method names
• Mixing variable assignments

3. Given this code, what will be the output?

facts = ['Paris is capital of France', 'France is in Europe']
answer = reasoner.use(facts)
print(answer)

Assuming reasoner.use() combines facts into a summary sentence.

medium

A. "Paris is capital of France and France is in Europe."

B. "Paris is capital of Europe."

C. "France is capital of Paris."

D. "Europe is in France."

Solution

1. Step 1: Understand input facts

   Facts list contains two true statements about Paris and France.

2. Step 2: Reasoner combines facts

   Reasoner merges facts into a combined sentence preserving meaning.

3. Final Answer:

   "Paris is capital of France and France is in Europe." -> Option A

4. Quick Check:

   Combined facts form correct summary [OK]

Hint: Look for combined true facts in output [OK]

Common Mistakes:

• Mixing up place names
• Ignoring fact order
• Assuming reasoner changes facts
• Choosing unrelated sentences

4. Identify the error in this code snippet combining retrieval and reasoning:

facts = reasoner.get_facts(query)
answer = retriever.use(facts)
print(answer)

medium

A. Variables facts and answer are swapped.

B. The print statement is missing parentheses.

C. Retriever should get facts, not reasoner.

D. No error; code runs correctly.

Solution

1. Step 1: Check roles of components

   Retriever is responsible for getting facts from query.

2. Step 2: Identify misuse

   Code wrongly calls reasoner.get_facts instead of retriever.get_facts.

3. Final Answer:

   Retriever should get facts, not reasoner. -> Option C

4. Quick Check:

   Retriever gets facts first [OK]

Hint: Retriever finds facts; reasoner uses them [OK]

Common Mistakes:

• Confusing retriever and reasoner roles
• Ignoring method names
• Assuming print syntax error
• Thinking variables are swapped

5. You want an AI agent to answer questions about a large document collection. Which approach best combines retrieval with reasoning to improve answer quality?

hard

A. Use only a reasoner without any retrieval step.

B. Use a reasoner to guess answers, then a retriever to check if facts exist.

C. Use only a retriever to return raw documents as answers.

D. Use a retriever to find relevant document parts, then a reasoner to synthesize an answer from those parts.

Solution

1. Step 1: Understand retrieval role

   Retriever finds relevant parts from large documents to reduce search space.

2. Step 2: Understand reasoning role

   Reasoner uses retrieved parts to create a clear, accurate answer.

3. Step 3: Evaluate options

   Use a retriever to find relevant document parts, then a reasoner to synthesize an answer from those parts. correctly sequences retrieval then reasoning for best quality.

4. Final Answer:

   Use a retriever to find relevant document parts, then a reasoner to synthesize an answer from those parts. -> Option D

5. Quick Check:

   Retrieve first, then reason [OK]

Hint: Retrieve relevant info first, then reason for answer [OK]

Common Mistakes:

• Reversing retrieval and reasoning order
• Skipping retrieval step
• Using only raw documents as answers
• Relying on guessing without facts