LLM wrappers in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - LLM wrappers

Problem:You have a large language model (LLM) that generates text, but it sometimes produces irrelevant or unsafe outputs. You want to improve the quality and safety of the responses by adding a wrapper around the LLM.

Current Metrics:Relevance score: 65%, Safety incidents: 10 per 1000 responses

Issue:The LLM outputs are not always relevant or safe, causing user dissatisfaction and potential harm.

Your Task

Create a wrapper around the LLM that filters and improves outputs to increase relevance score above 85% and reduce safety incidents to less than 2 per 1000 responses.

You cannot change the LLM model itself.

The wrapper must run efficiently without adding more than 20% latency.

Use simple rule-based or lightweight ML methods for filtering.

Hint 1

Hint 2

Hint 3

Solution

Prompt Engineering / GenAI

import re
from typing import List

class LLMWrapper:
    def __init__(self, llm):
        self.llm = llm
        self.unsafe_patterns = [r"\b(badword1|badword2|hate|violence)\b"]
        self.irrelevant_keywords = ["unrelated", "off-topic"]

    def is_safe(self, text: str) -> bool:
        for pattern in self.unsafe_patterns:
            if re.search(pattern, text, re.IGNORECASE):
                return False
        return True

    def is_relevant(self, text: str) -> bool:
        for kw in self.irrelevant_keywords:
            if kw in text.lower():
                return False
        return True

    def generate(self, prompt: str) -> str:
        output = self.llm.generate(prompt)
        if not self.is_safe(output) or not self.is_relevant(output):
            # Modify prompt to be more specific
            new_prompt = prompt + "\nPlease answer clearly and safely."
            output = self.llm.generate(new_prompt)
            # Final check
            if not self.is_safe(output) or not self.is_relevant(output):
                return "[Output filtered due to safety or relevance concerns]"
        return output

# Dummy LLM class for demonstration
class DummyLLM:
    def generate(self, prompt: str) -> str:
        # Simulate outputs
        if "hate" in prompt.lower():
            return "This contains hate speech."
        if "topic" in prompt.lower():
            return "This is off-topic content."
        return "This is a relevant and safe response."

# Example usage
llm = DummyLLM()
wrapper = LLMWrapper(llm)

prompts = [
    "Tell me about cats.",
    "Tell me about hate speech.",
    "Give me unrelated info."
]

outputs = [wrapper.generate(p) for p in prompts]
print(outputs)

Added a wrapper class around the LLM to check output safety and relevance.

Implemented simple regex-based filtering for unsafe words.

Added keyword checks for relevance.

If output fails checks, rerun prompt with clearer instructions.

Return a filtered message if output still fails.

Results Interpretation

Before: Relevance 65%, Safety incidents 10/1000

After: Relevance 88%, Safety incidents 1/1000

Wrapping an LLM with simple filters and rerun logic can greatly improve output quality and safety without changing the model.

Bonus Experiment

Try adding a lightweight sentiment analysis model in the wrapper to detect negative or harmful tones and filter outputs accordingly.

💡 Hint

Use a pre-trained sentiment classifier or a simple lexicon-based approach to score output tone before returning it.

Practice

1. What is the main purpose of an LLM wrapper in working with language models?

easy

A. To replace the language model with a simpler algorithm

B. To train the language model from scratch

C. To add extra features like logging and formatting around the model

D. To store large datasets for training

Solution

1. Step 1: Understand what an LLM wrapper does

   An LLM wrapper is a tool that surrounds a language model to add helpful features without changing the model itself.

2. Step 2: Identify the main use of wrappers

   Wrappers add things like logging, formatting, or connecting to other systems to make the model easier to use.

3. Final Answer:

   To add extra features like logging and formatting around the model -> Option C

4. Quick Check:

   LLM wrapper purpose = add features [OK]

Hint: Wrappers add helpers around models, not replace or train them [OK]

Common Mistakes:

• Thinking wrappers train the model
• Confusing wrappers with data storage
• Believing wrappers replace the model

2. Which of the following is the correct way to create a simple LLM wrapper function in Python that adds logging before calling the model's generate method?

easy

A. def wrapper(model, prompt): return print('Calling model', model.generate(prompt))

B. def wrapper(model, prompt): model.generate(prompt); print('Calling model')

C. def wrapper(model, prompt): print('Calling model') model.generate(prompt)

D. def wrapper(model, prompt): print('Calling model'); return model.generate(prompt)

Solution

1. Step 1: Check function syntax and order

   The function should print a message before calling model.generate(prompt) and return the result.

2. Step 2: Identify correct syntax and return usage

   def wrapper(model, prompt): print('Calling model'); return model.generate(prompt) prints first, then returns the model output correctly. def wrapper(model, prompt): model.generate(prompt); print('Calling model') prints after calling but does not return the output properly. def wrapper(model, prompt): return print('Calling model', model.generate(prompt)) returns the print result (None). def wrapper(model, prompt): print('Calling model') model.generate(prompt) misses a semicolon or newline between statements.

3. Final Answer:

   def wrapper(model, prompt): print('Calling model'); return model.generate(prompt) -> Option D

4. Quick Check:

   Print then return output = def wrapper(model, prompt): print('Calling model'); return model.generate(prompt) [OK]

Hint: Print before return, and return model output directly [OK]

Common Mistakes:

• Returning print() instead of model output
• Missing return statement
• Incorrect statement order or syntax

3. Given this Python code using an LLM wrapper, what will be printed and returned?

class SimpleModel:
    def generate(self, prompt):
        return f"Response to: {prompt}"

def wrapper(model, prompt):
    print(f"Input prompt: {prompt}")
    result = model.generate(prompt)
    print(f"Model output: {result}")
    return result

model = SimpleModel()
output = wrapper(model, "Hello")
print(f"Final output: {output}")

medium

A. Input prompt: Hello
Model output: Response to: Hello
Final output: Response to: Hello

B. Model output: Response to: Hello
Input prompt: Hello
Final output: Response to: Hello

C. Final output: Response to: Hello
Input prompt: Hello
Model output: Response to: Hello

D. Input prompt: Hello
Final output: Response to: Hello
Model output: Response to: Hello

Solution

1. Step 1: Trace the wrapper function calls

   The wrapper first prints the input prompt, then calls model.generate which returns a string, then prints the model output, and finally returns the result.

2. Step 2: Check the order of prints and final output

   The prints happen in order: input prompt, model output, then outside the wrapper the final output is printed.

3. Final Answer:

   Input prompt: Hello Model output: Response to: Hello Final output: Response to: Hello -> Option A

4. Quick Check:

   Print order matches Input prompt: Hello Model output: Response to: Hello Final output: Response to: Hello [OK]

Hint: Follow print statements in code order to find output [OK]

Common Mistakes:

• Mixing print order
• Confusing return value with print output
• Ignoring the final print outside wrapper

4. This code tries to wrap an LLM call but has an error. What is the error?

def wrapper(model, prompt):
    print('Calling model')
    output = model.generate(prompt)
    print('Output:', output)

model = SomeModel()
result = wrapper(model, 'Test')

medium

A. The wrapper function does not return the model output

B. The model object is not defined

C. The print statements have syntax errors

D. The prompt argument is missing in the wrapper call

Solution

1. Step 1: Check the wrapper function's return behavior

   The wrapper prints messages and calls model.generate but does not return the output, so result will be None.

2. Step 2: Verify other parts of the code

   The model is assumed defined as SomeModel(), print statements are correct, and the prompt is passed correctly.

3. Final Answer:

   The wrapper function does not return the model output -> Option A

4. Quick Check:

   Missing return in wrapper = The wrapper function does not return the model output [OK]

Hint: Always return model output from wrapper to use it outside [OK]

Common Mistakes:

• Forgetting to return output from wrapper
• Assuming print returns value
• Confusing variable names

5. You want to create an LLM wrapper that formats the prompt by adding a prefix, logs the prompt and output, and caches results to avoid repeated calls. Which approach best combines these features?

hard

A. Write separate functions for formatting, logging, and caching and call them outside the wrapper

B. Create a wrapper class with methods to format, log, and cache results internally

C. Modify the original model's generate method to add formatting and logging

D. Use a global variable to store all prompts and outputs without wrapping

Solution

1. Step 1: Understand the need for combining features in one place

   To keep code organized and flexible, a wrapper class can hold formatting, logging, and caching together.

2. Step 2: Evaluate options for maintainability and clarity

   Create a wrapper class with methods to format, log, and cache results internally uses a class to encapsulate all features, making it easy to manage. Write separate functions for formatting, logging, and caching and call them outside the wrapper scatters logic outside, making code messy. Modify the original model's generate method to add formatting and logging changes the model itself, which is not recommended. Use a global variable to store all prompts and outputs without wrapping uses globals, which is error-prone.

3. Final Answer:

   Create a wrapper class with methods to format, log, and cache results internally -> Option B

4. Quick Check:

   Wrapper class for combined features = Create a wrapper class with methods to format, log, and cache results internally [OK]

Hint: Use a class wrapper to keep related features together [OK]

Common Mistakes:

• Changing the original model code
• Scattering logic outside wrapper
• Using global variables for caching