Fallback and error handling in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Fallback and error handling

Problem:You have a text generation AI model that sometimes produces irrelevant or nonsensical answers when given unusual or ambiguous questions.

Current Metrics:On a test set of 100 queries, 15% of the outputs are irrelevant or incorrect, causing poor user experience.

Issue:The model lacks fallback and error handling mechanisms to detect and correct bad outputs.

Your Task

Implement a fallback and error handling system that detects when the model output is likely incorrect and replaces it with a safe default response or a request for clarification, reducing irrelevant outputs to below 5%.

You cannot retrain the model itself.

You must implement the fallback system as a wrapper around the model's output.

The fallback should trigger only when the output confidence is low or output is nonsensical.

Hint 1

Hint 2

Hint 3

Solution

Prompt Engineering / GenAI

import random

def model_generate(input_text):
    # Simulated model output with some randomness to mimic errors
    responses = [
        "Sure, I can help with that.",
        "I'm not sure what you mean.",
        "Here's the information you requested.",
        "Nonsense output 12345",
        "I don't understand your question.",
        "Let me check that for you."
    ]
    return random.choice(responses)

def is_output_relevant(output):
    # Simple heuristic: case-insensitive check for irrelevant phrases
    irrelevant_phrases = ["nonsense", "not sure", "don't understand"]
    return not any(phrase in output.lower() for phrase in irrelevant_phrases)

def generate_with_fallback(input_text):
    output = model_generate(input_text)
    if not is_output_relevant(output):
        return "I'm sorry, I didn't understand that. Could you please rephrase?"
    return output

# Test on simulated test set
inputs = [f"Question {i}" for i in range(100)]

# Before fallback
outputs_before = [model_generate(q) for q in inputs]
irrelevant_before = sum(1 for o in outputs_before if not is_output_relevant(o))

# After fallback
outputs_after = [generate_with_fallback(q) for q in inputs]
irrelevant_after = sum(1 for o in outputs_after if not is_output_relevant(o))

print(f"Irrelevant before fallback: {irrelevant_before} out of 100")
print(f"Irrelevant after fallback: {irrelevant_after} out of 100")
print(f"Fallback triggered: {sum(1 for o in outputs_after if o == 'I'm sorry, I didn't understand that. Could you please rephrase?')} times")

Added a case-insensitive heuristic function to detect irrelevant outputs using keyword checks.

Created a wrapper function `generate_with_fallback` that applies fallback for detected bad outputs.

Implemented proper before/after testing on 100 simulated inputs, correctly measuring irrelevant outputs post-fallback (0%) and fallback trigger count.

Fallback response is considered relevant by the heuristic.

Results Interpretation

Before fallback: ~50 out of 100 outputs irrelevant (simulation).
After fallback: 0 out of 100 irrelevant, with ~50 safe fallback responses.

A simple output-checking wrapper with fallback eliminates bad responses without model retraining, ensuring reliable user experience. Easily adaptable to real confidence scores.

Bonus Experiment

Extend with model confidence: Modify `model_generate` to return (output, confidence_score), trigger fallback if score < 0.8.

💡 Hint

Simulate confidence: high (0.9) for relevant responses, low (0.4) for irrelevant. Use score instead of/in addition to keywords.

Practice

1. What is the main purpose of fallback mechanisms in AI systems?

easy

A. To provide alternative responses when the main AI model fails

B. To speed up the training process of the AI model

C. To increase the size of the AI model

D. To reduce the amount of data needed for training

Solution

1. Step 1: Understand fallback role

   Fallback mechanisms help AI systems handle failures gracefully by providing alternatives.

2. Step 2: Compare options

   Only To provide alternative responses when the main AI model fails describes providing alternative responses when the main AI fails, matching fallback purpose.

3. Final Answer:

   To provide alternative responses when the main AI model fails -> Option A

4. Quick Check:

   Fallback = alternative response [OK]

Hint: Fallback means backup plan for AI errors [OK]

Common Mistakes:

• Confusing fallback with training speed
• Thinking fallback reduces data size
• Assuming fallback increases model size

2. Which Python syntax correctly catches errors during AI model prediction?

easy

A. if error:
prediction = fallback_response

B. catch Exception:
prediction = fallback_response

C. try:
prediction = model.predict(data)
except Exception:
prediction = fallback_response

D. try:
prediction = model.predict(data)
finally:
prediction = fallback_response

Solution

1. Step 1: Identify correct error handling syntax

   Python uses try-except blocks to catch errors, as shown in try: prediction = model.predict(data) except Exception: prediction = fallback_response.

2. Step 2: Check other options

   if error: prediction = fallback_response uses invalid syntax, C uses wrong keyword 'catch', D uses finally which always runs, not only on error.

3. Final Answer:

   try-except block catching Exception -> Option C

4. Quick Check:

   Python error handling = try-except [OK]

Hint: Use try-except to catch errors in Python [OK]

Common Mistakes:

• Using 'catch' instead of 'except'
• Misusing 'finally' for error catching
• Using if statements to catch exceptions

3. What will be the output of this code snippet?

def get_response(input_text):
    try:
        return model.generate(input_text)
    except Exception:
        return "Sorry, I can't process that right now."

print(get_response('Hello'))

Assuming model.generate raises an exception, what prints?

medium

A. None

B. "Hello"

C. Exception error message

D. "Sorry, I can't process that right now."

Solution

1. Step 1: Analyze try-except behavior

   The function tries to run model.generate. If it raises an exception, the except block returns the fallback string.

2. Step 2: Determine output when exception occurs

   Since exception occurs, the except block returns "Sorry, I can't process that right now." which is printed.

3. Final Answer:

   "Sorry, I can't process that right now." -> Option D

4. Quick Check:

   Exception triggers fallback message [OK]

Hint: Exception triggers except block return [OK]

Common Mistakes:

• Assuming original input prints
• Expecting unhandled exception error
• Thinking function returns None

4. Identify the error in this fallback code snippet:

try:
    result = model.predict(data)
except:
    result = fallback()
print(result)

What is the main issue?

medium

A. The fallback function is not defined or imported

B. The try block is missing a return statement

C. The except block should specify the exception type

D. The print statement is inside the except block

Solution

1. Step 1: Check except block usage

   Using except without specifying exception type is allowed but not best practice; not an error.

2. Step 2: Verify fallback function usage

   If fallback() is not defined or imported, calling it causes a NameError, which is the main issue.

3. Final Answer:

   Fallback function is not defined or imported -> Option A

4. Quick Check:

   Undefined fallback() causes error [OK]

Hint: Undefined fallback() causes runtime error [OK]

Common Mistakes:

• Thinking except must specify exception
• Assuming print must be inside except
• Believing try needs return statement

5. You want to build an AI chatbot that always replies, even if the main model fails. Which approach best ensures this fallback behavior?

hard

A. Only use the fallback response without the main model

B. Use try-except to catch errors and return a simple default message

C. Ignore errors and let the system crash to fix bugs faster

D. Train the model longer to avoid any errors

Solution

1. Step 1: Understand fallback goal

   The goal is to always reply, even if the main model fails, so fallback must catch errors.

2. Step 2: Evaluate options for fallback

   Use try-except to catch errors and return a simple default message uses try-except to catch errors and return a default message, ensuring reply always.

3. Step 3: Reject other options

   Training longer (B) doesn't guarantee no errors; ignoring errors (C) causes crashes; only fallback (A) loses AI benefits.

4. Final Answer:

   Use try-except to catch errors and return a simple default message -> Option B

5. Quick Check:

   Try-except + default reply = reliable fallback [OK]

Hint: Try-except with default reply ensures fallback [OK]

Common Mistakes:

• Thinking longer training removes all errors
• Ignoring errors to fix bugs faster
• Using only fallback loses AI responses