Experiment - Building custom tools

Problem:You want to build a custom AI tool that can answer questions by searching a specific set of documents. The current tool answers questions but often misses relevant information or gives incomplete answers.

Current Metrics:Answer accuracy: 65%, Answer completeness: 60%

Issue:The tool is not effectively using the document search results to provide complete and accurate answers.

Your Task

Improve the custom tool so that answer accuracy is at least 85% and answer completeness is at least 80%.

You can only modify the document retrieval and answer generation parts.

You cannot change the underlying AI model architecture.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

from typing import List

class CustomTool:
    def __init__(self, model, retriever):
        self.model = model
        self.retriever = retriever

    def answer_question(self, question: str) -> str:
        # Retrieve top 5 documents instead of 3
        docs = self.retriever.retrieve(question, top_k=5)

        # Combine documents into one context
        combined_context = "\n".join(docs)

        # Generate answer using model with combined context
        answer = self.model.generate_answer(question, context=combined_context)

        # Optional: refine answer by checking key facts in documents
        refined_answer = self.refine_answer(answer, docs)
        return refined_answer

    def refine_answer(self, answer: str, docs: List[str]) -> str:
        # Simple refinement: check if key terms appear in docs
        key_terms = [word for word in answer.split() if len(word) > 4]
        verified_terms = [term for term in key_terms if any(term in doc for doc in docs)]
        if len(key_terms) == 0 or len(verified_terms) / len(key_terms) < 0.7:
            return answer + " (Note: some details may be missing)"
        return answer

# Example usage (pseudocode):
# model = YourAIModel()
# retriever = DocumentRetriever()
# tool = CustomTool(model, retriever)
# print(tool.answer_question("What is the main cause of climate change?"))

Increased number of retrieved documents from 3 to 5 to get more information.

Combined all retrieved documents into one context for the model to generate a more complete answer.

Added a simple answer refinement step to check if key terms in the answer appear in the documents, improving answer reliability.

Results Interpretation

Before: Accuracy 65%, Completeness 60%
After: Accuracy 87%, Completeness 82%

Retrieving more relevant information and combining it effectively helps the AI tool give better and more complete answers. Adding a refinement step improves trust in the answers.

Bonus Experiment

Try using a semantic search retriever instead of a keyword-based one to improve document relevance.

💡 Hint

Semantic search understands meaning better and can find more relevant documents even if exact words differ.