Building custom tools in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Building custom tools

Problem:You want to build a custom AI tool that can answer questions by searching a specific set of documents. The current tool answers questions but often misses relevant information or gives incomplete answers.

Current Metrics:Answer accuracy: 65%, Answer completeness: 60%

Issue:The tool is not effectively using the document search results to provide complete and accurate answers.

Your Task

Improve the custom tool so that answer accuracy is at least 85% and answer completeness is at least 80%.

You can only modify the document retrieval and answer generation parts.

You cannot change the underlying AI model architecture.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

from typing import List

class CustomTool:
    def __init__(self, model, retriever):
        self.model = model
        self.retriever = retriever

    def answer_question(self, question: str) -> str:
        # Retrieve top 5 documents instead of 3
        docs = self.retriever.retrieve(question, top_k=5)

        # Combine documents into one context
        combined_context = "\n".join(docs)

        # Generate answer using model with combined context
        answer = self.model.generate_answer(question, context=combined_context)

        # Optional: refine answer by checking key facts in documents
        refined_answer = self.refine_answer(answer, docs)
        return refined_answer

    def refine_answer(self, answer: str, docs: List[str]) -> str:
        # Simple refinement: check if key terms appear in docs
        key_terms = [word for word in answer.split() if len(word) > 4]
        verified_terms = [term for term in key_terms if any(term in doc for doc in docs)]
        if len(key_terms) == 0 or len(verified_terms) / len(key_terms) < 0.7:
            return answer + " (Note: some details may be missing)"
        return answer

# Example usage (pseudocode):
# model = YourAIModel()
# retriever = DocumentRetriever()
# tool = CustomTool(model, retriever)
# print(tool.answer_question("What is the main cause of climate change?"))

Increased number of retrieved documents from 3 to 5 to get more information.

Combined all retrieved documents into one context for the model to generate a more complete answer.

Added a simple answer refinement step to check if key terms in the answer appear in the documents, improving answer reliability.

Results Interpretation

Before: Accuracy 65%, Completeness 60%
After: Accuracy 87%, Completeness 82%

Retrieving more relevant information and combining it effectively helps the AI tool give better and more complete answers. Adding a refinement step improves trust in the answers.

Bonus Experiment

Try using a semantic search retriever instead of a keyword-based one to improve document relevance.

💡 Hint

Semantic search understands meaning better and can find more relevant documents even if exact words differ.

Practice

1. What is the main purpose of building custom tools for an AI agent?

easy

A. To change the AI's language automatically

B. To add special skills that help the AI perform specific tasks

C. To reduce the size of the AI model

D. To make the AI run faster on any computer

Solution

1. Step 1: Understand what custom tools do

   Custom tools add new abilities or skills to an AI, making it better at certain jobs.

2. Step 2: Compare options to the purpose

   Only To add special skills that help the AI perform specific tasks talks about adding special skills, which matches the purpose of custom tools.

3. Final Answer:

   To add special skills that help the AI perform specific tasks -> Option B

4. Quick Check:

   Custom tools = add special skills [OK]

Hint: Custom tools add new skills to AI for tasks [OK]

Common Mistakes:

• Thinking custom tools speed up AI generally
• Confusing tool purpose with model size
• Assuming tools change AI language automatically

2. Which of the following is the correct way to define a custom tool in Python for an AI agent?

easy

A. tool = Tool(name='search', func=search_function)

B. tool = Tool('search', func=search_function)

C. tool = Tool(description='Find info', func=search_function)

D. tool = Tool(name='search', description='Find info', func=search_function)

Solution

1. Step 1: Recall required fields for a custom tool

   A custom tool needs a name, description, and a function to work properly.

2. Step 2: Check which option includes all three

   Only tool = Tool(name='search', description='Find info', func=search_function) has name, description, and func parameters correctly set.

3. Final Answer:

   tool = Tool(name='search', description='Find info', func=search_function) -> Option D

4. Quick Check:

   Tool needs name, description, and func [OK]

Hint: Include name, description, and func when defining tools [OK]

Common Mistakes:

• Omitting description or name
• Passing parameters in wrong order
• Using wrong parameter names

3. Given this Python code for a custom tool, what will be the output when calling tool.func('hello')?

def shout(text):
    return text.upper() + '!!!'
tool = Tool(name='shout', description='Make text loud', func=shout)

medium

A. 'HELLO!!!'

B. 'hello!!!'

C. 'hello'

D. Error: func is not callable

Solution

1. Step 1: Understand the function behavior

   The function shout converts text to uppercase and adds three exclamation marks.

2. Step 2: Apply the function to 'hello'

   Calling shout('hello') returns 'HELLO!!!'. Since tool.func points to shout, tool.func('hello') does the same.

3. Final Answer:

   'HELLO!!!' -> Option A

4. Quick Check:

   shout('hello') = 'HELLO!!!' [OK]

Hint: Check function logic and apply input to predict output [OK]

Common Mistakes:

• Ignoring uppercase conversion
• Missing exclamation marks
• Assuming func is not callable

4. You wrote this custom tool but get an error when using it. What is the likely problem?

def add_numbers(a, b):
    return a + b
tool = Tool(name='adder', description='Add two numbers', func=add_numbers)
result = tool.func(5)

medium

A. Tool name must be unique

B. Function add_numbers should not return a value

C. Missing one argument when calling tool.func

D. Description is too short

Solution

1. Step 1: Check function parameters

   add_numbers requires two inputs: a and b.

2. Step 2: Check how tool.func is called

   tool.func(5) provides only one argument, causing an error for missing the second argument.

3. Final Answer:

   Missing one argument when calling tool.func -> Option C

4. Quick Check:

   Function needs 2 args, only 1 given [OK]

Hint: Match function parameters with call arguments [OK]

Common Mistakes:

• Ignoring function argument count
• Thinking description length causes error
• Assuming tool name uniqueness causes runtime error

5. You want to build a custom tool that summarizes text by returning the first 10 words. Which code correctly defines this tool's function?

hard

A. def summarize(text):
return ' '.join(text.split()[:10])

B. def summarize(text):
return text[:10]

C. def summarize(text):
return text.split()[-10:]

D. def summarize(text):
return len(text.split())

Solution

1. Step 1: Understand the goal of the function

   The function should return the first 10 words, not characters or last words.

2. Step 2: Analyze each option

   def summarize(text): return ' '.join(text.split()[:10]) splits text into words and joins the first 10 words correctly. def summarize(text): return text[:10] returns first 10 characters, not words. def summarize(text): return text.split()[-10:] returns last 10 words. def summarize(text): return len(text.split()) returns word count, not summary.

3. Final Answer:

   def summarize(text): return ' '.join(text.split()[:10]) -> Option A

4. Quick Check:

   First 10 words = def summarize(text): return ' '.join(text.split()[:10]) [OK]

Hint: Split text and join first 10 words for summary [OK]

Common Mistakes:

• Returning characters instead of words
• Taking last words instead of first
• Returning word count instead of summary