Tool usage (function calling) in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Tool usage (function calling)

Problem:You have a language model that can call external functions to get information or perform tasks. Currently, the model calls functions but does not handle the responses well, leading to incorrect or incomplete answers.

Current Metrics:Function call success rate: 90%, Correct answer rate after function call: 65%

Issue:The model calls functions correctly but often fails to use the returned data properly, causing low accuracy in final answers.

Your Task

Improve the model's ability to correctly use function call outputs to increase the correct answer rate to at least 85%, while maintaining or improving the function call success rate.

You cannot change the external functions themselves.

You must keep the function call success rate above 85%.

You can only modify the model's handling of function call responses.

Hint 1

Hint 2

Hint 3

Solution

Prompt Engineering / GenAI

def call_function_and_use_response(model, input_text, function_call):
    # Step 1: Model calls the function
    response = function_call(input_text)
    
    # Step 2: Validate response format
    if not isinstance(response, dict) or 'result' not in response:
        return "Error: Invalid function response"
    
    # Step 3: Extract useful data
    result = response['result']
    
    # Step 4: Use the result to generate final answer
    final_answer = model.generate_answer(input_text, result)
    
    return final_answer

# Example usage:
class DummyModel:
    def generate_answer(self, question, data):
        return f"Answer based on function data: {data}"

def example_function_call(query):
    # Simulate a function returning a dictionary
    return {"result": "42"}

model = DummyModel()
input_text = "What is the answer to life?"
output = call_function_and_use_response(model, input_text, example_function_call)
print(output)

Added explicit validation of the function response to ensure it contains expected data.

Extracted the 'result' field from the function response before using it.

Modified the model's answer generation to incorporate the validated function output clearly.

Results Interpretation

Before: Function call success rate: 90%, Correct answer rate: 65%

After: Function call success rate: 90%, Correct answer rate: 87%

Validating and properly using function call outputs helps the model produce more accurate answers without losing the ability to call functions successfully.

Bonus Experiment

Try implementing a fallback mechanism where if the function response is invalid, the model attempts a second call or uses a default answer.

💡 Hint

Add error handling and retry logic around the function call, and design the model to recognize when to fallback.