Why evaluation prevents production failures in LangChain - See It in Action

Why Evaluation Prevents Production Failures in Langchain

📖 Scenario: You are building a simple Langchain application that uses an LLM to answer questions. To avoid failures in production, you want to evaluate the LLM's responses on a small test set before deploying.

🎯 Goal: Build a Langchain script that sets up test data, configures an evaluation threshold, runs the evaluation on sample inputs, and adds a final check to prevent deployment if the evaluation score is too low.

📋 What You'll Learn

Create a dictionary called test_data with exact question-answer pairs

Add a variable called min_accuracy set to 0.8

Write a function evaluate_model that compares model answers to expected answers and returns accuracy

Add a final check that raises an exception if accuracy is below min_accuracy

💡 Why This Matters

🌍 Real World

Evaluating AI models before deployment helps catch errors early and avoid bad user experiences or system failures.

💼 Career

Many AI and software engineering roles require writing tests and evaluation scripts to ensure quality and reliability before production release.

1

DATA SETUP: Create test data for evaluation

Create a dictionary called test_data with these exact entries: 'What is the capital of France?': 'Paris', 'What color is the sky?': 'Blue', and 'How many legs does a spider have?': '8'.

LangChain

# Create the test_data dictionary with exact question-answer pairs
# Your code here

Need a hint?

Use a Python dictionary with exact keys and values as shown.

2

CONFIGURATION: Set minimum accuracy threshold

Add a variable called min_accuracy and set it to 0.8 to represent the minimum acceptable accuracy for the evaluation.

LangChain

test_data = {
    'What is the capital of France?': 'Paris',
    'What color is the sky?': 'Blue',
    'How many legs does a spider have?': '8'
}
# Set the minimum accuracy threshold
# Your code here

Need a hint?

Just create a variable named min_accuracy and assign it the value 0.8.

3

CORE LOGIC: Write evaluation function

Write a function called evaluate_model that takes a model function and the test_data dictionary. It should return the accuracy as the fraction of correct answers. Use a for question, expected_answer in test_data.items() loop and compare the model's answer to the expected answer.

LangChain

test_data = {
    'What is the capital of France?': 'Paris',
    'What color is the sky?': 'Blue',
    'How many legs does a spider have?': '8'
}

min_accuracy = 0.8

# Define evaluate_model function below
# Your code here

Need a hint?

Loop over test_data.items(), call model_func(question), count matches, then return accuracy.

4

COMPLETION: Add final evaluation check before deployment

Add code that calls evaluate_model with a dummy model_func that returns answers from test_data. Then check if the returned accuracy is less than min_accuracy. If so, raise an Exception with the message 'Model accuracy too low for deployment'.

LangChain

test_data = {
    'What is the capital of France?': 'Paris',
    'What color is the sky?': 'Blue',
    'How many legs does a spider have?': '8'
}

min_accuracy = 0.8

def evaluate_model(model_func, test_data):
    correct = 0
    total = len(test_data)
    for question, expected_answer in test_data.items():
        answer = model_func(question)
        if answer == expected_answer:
            correct += 1
    return correct / total

# Add final evaluation check below
# Your code here

Need a hint?

Use a lambda to simulate the model, call evaluate_model, then raise an exception if accuracy is below min_accuracy.