LLM wrappers in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - LLM wrappers

Which metric matters for LLM wrappers and WHY

LLM wrappers are tools that help use large language models (LLMs) better. The key metric to check is response accuracy, which means how correct or relevant the LLM's answers are when wrapped. We also look at latency (speed) and robustness (handling different inputs well). Accuracy matters most because the wrapper should keep or improve the LLM's quality. Speed matters because users want quick answers. Robustness ensures the wrapper does not break or give bad results on tricky inputs.

Confusion matrix or equivalent visualization

For LLM wrappers, we often check classification or question-answering tasks. Here is an example confusion matrix for a classification task after wrapping an LLM:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP): 80 | False Positive (FP): 10 |
      | False Negative (FN): 20 | True Negative (TN): 90 |
    

This shows how many answers the wrapped LLM got right or wrong. We use these numbers to calculate precision and recall.

Precision vs Recall tradeoff with examples

Precision means how many answers the wrapper marked as correct really are correct. Recall means how many of all correct answers the wrapper found.

For example, if the wrapper is used in a customer support chatbot, high precision is important so it does not give wrong info. But if used in medical advice, high recall is more important to catch all possible issues.

Improving precision may lower recall and vice versa. The wrapper design should balance these based on the use case.

What "good" vs "bad" metric values look like for LLM wrappers

Good values:

• Precision and recall above 85% for classification tasks
• Low latency (under 1 second response time)
• Stable results across different inputs (robustness)

Bad values:

• Precision or recall below 50%, meaning many wrong or missed answers
• High latency causing slow responses
• Unstable or inconsistent outputs on similar inputs

Common pitfalls in metrics for LLM wrappers

• Accuracy paradox: High overall accuracy but poor performance on important classes.
• Data leakage: Wrapper accidentally uses test data during tuning, inflating metrics.
• Overfitting: Wrapper tuned too much on training data, fails on new inputs.
• Ignoring latency: Focusing only on accuracy but wrapper slows down user experience.
• Not measuring robustness: Wrapper fails silently on unusual inputs.

Self-check question

Your LLM wrapper model has 98% accuracy but only 12% recall on fraud detection. Is it good for production? Why or why not?

Answer: No, it is not good. Even though accuracy is high, the very low recall means the wrapper misses most fraud cases. In fraud detection, missing fraud is very risky. The model should have high recall to catch as many frauds as possible, even if accuracy is slightly lower.

Key Result

For LLM wrappers, balancing high precision, recall, and low latency ensures accurate, fast, and reliable outputs.

Practice

1. What is the main purpose of an LLM wrapper in working with language models?

easy

A. To replace the language model with a simpler algorithm

B. To train the language model from scratch

C. To add extra features like logging and formatting around the model

D. To store large datasets for training

Solution

1. Step 1: Understand what an LLM wrapper does

   An LLM wrapper is a tool that surrounds a language model to add helpful features without changing the model itself.

2. Step 2: Identify the main use of wrappers

   Wrappers add things like logging, formatting, or connecting to other systems to make the model easier to use.

3. Final Answer:

   To add extra features like logging and formatting around the model -> Option C

4. Quick Check:

   LLM wrapper purpose = add features [OK]

Hint: Wrappers add helpers around models, not replace or train them [OK]

Common Mistakes:

• Thinking wrappers train the model
• Confusing wrappers with data storage
• Believing wrappers replace the model

2. Which of the following is the correct way to create a simple LLM wrapper function in Python that adds logging before calling the model's generate method?

easy

A. def wrapper(model, prompt): return print('Calling model', model.generate(prompt))

B. def wrapper(model, prompt): model.generate(prompt); print('Calling model')

C. def wrapper(model, prompt): print('Calling model') model.generate(prompt)

D. def wrapper(model, prompt): print('Calling model'); return model.generate(prompt)

Solution

1. Step 1: Check function syntax and order

   The function should print a message before calling model.generate(prompt) and return the result.

2. Step 2: Identify correct syntax and return usage

   def wrapper(model, prompt): print('Calling model'); return model.generate(prompt) prints first, then returns the model output correctly. def wrapper(model, prompt): model.generate(prompt); print('Calling model') prints after calling but does not return the output properly. def wrapper(model, prompt): return print('Calling model', model.generate(prompt)) returns the print result (None). def wrapper(model, prompt): print('Calling model') model.generate(prompt) misses a semicolon or newline between statements.

3. Final Answer:

   def wrapper(model, prompt): print('Calling model'); return model.generate(prompt) -> Option D

4. Quick Check:

   Print then return output = def wrapper(model, prompt): print('Calling model'); return model.generate(prompt) [OK]

Hint: Print before return, and return model output directly [OK]

Common Mistakes:

• Returning print() instead of model output
• Missing return statement
• Incorrect statement order or syntax

3. Given this Python code using an LLM wrapper, what will be printed and returned?

class SimpleModel:
    def generate(self, prompt):
        return f"Response to: {prompt}"

def wrapper(model, prompt):
    print(f"Input prompt: {prompt}")
    result = model.generate(prompt)
    print(f"Model output: {result}")
    return result

model = SimpleModel()
output = wrapper(model, "Hello")
print(f"Final output: {output}")

medium

A. Input prompt: Hello
Model output: Response to: Hello
Final output: Response to: Hello

B. Model output: Response to: Hello
Input prompt: Hello
Final output: Response to: Hello

C. Final output: Response to: Hello
Input prompt: Hello
Model output: Response to: Hello

D. Input prompt: Hello
Final output: Response to: Hello
Model output: Response to: Hello

Solution

1. Step 1: Trace the wrapper function calls

   The wrapper first prints the input prompt, then calls model.generate which returns a string, then prints the model output, and finally returns the result.

2. Step 2: Check the order of prints and final output

   The prints happen in order: input prompt, model output, then outside the wrapper the final output is printed.

3. Final Answer:

   Input prompt: Hello Model output: Response to: Hello Final output: Response to: Hello -> Option A

4. Quick Check:

   Print order matches Input prompt: Hello Model output: Response to: Hello Final output: Response to: Hello [OK]

Hint: Follow print statements in code order to find output [OK]

Common Mistakes:

• Mixing print order
• Confusing return value with print output
• Ignoring the final print outside wrapper

4. This code tries to wrap an LLM call but has an error. What is the error?

def wrapper(model, prompt):
    print('Calling model')
    output = model.generate(prompt)
    print('Output:', output)

model = SomeModel()
result = wrapper(model, 'Test')

medium

A. The wrapper function does not return the model output

B. The model object is not defined

C. The print statements have syntax errors

D. The prompt argument is missing in the wrapper call

Solution

1. Step 1: Check the wrapper function's return behavior

   The wrapper prints messages and calls model.generate but does not return the output, so result will be None.

2. Step 2: Verify other parts of the code

   The model is assumed defined as SomeModel(), print statements are correct, and the prompt is passed correctly.

3. Final Answer:

   The wrapper function does not return the model output -> Option A

4. Quick Check:

   Missing return in wrapper = The wrapper function does not return the model output [OK]

Hint: Always return model output from wrapper to use it outside [OK]

Common Mistakes:

• Forgetting to return output from wrapper
• Assuming print returns value
• Confusing variable names

5. You want to create an LLM wrapper that formats the prompt by adding a prefix, logs the prompt and output, and caches results to avoid repeated calls. Which approach best combines these features?

hard

A. Write separate functions for formatting, logging, and caching and call them outside the wrapper

B. Create a wrapper class with methods to format, log, and cache results internally

C. Modify the original model's generate method to add formatting and logging

D. Use a global variable to store all prompts and outputs without wrapping

Solution

1. Step 1: Understand the need for combining features in one place

   To keep code organized and flexible, a wrapper class can hold formatting, logging, and caching together.

2. Step 2: Evaluate options for maintainability and clarity

   Create a wrapper class with methods to format, log, and cache results internally uses a class to encapsulate all features, making it easy to manage. Write separate functions for formatting, logging, and caching and call them outside the wrapper scatters logic outside, making code messy. Modify the original model's generate method to add formatting and logging changes the model itself, which is not recommended. Use a global variable to store all prompts and outputs without wrapping uses globals, which is error-prone.

3. Final Answer:

   Create a wrapper class with methods to format, log, and cache results internally -> Option B

4. Quick Check:

   Wrapper class for combined features = Create a wrapper class with methods to format, log, and cache results internally [OK]

Hint: Use a class wrapper to keep related features together [OK]

Common Mistakes:

• Changing the original model code
• Scattering logic outside wrapper
• Using global variables for caching