Automated evaluation pipelines in LangChain - Mini Project: Build & Apply

Automated Evaluation Pipelines with LangChain

📖 Scenario: You are building a simple automated evaluation pipeline using LangChain to test how well a language model answers questions. This pipeline will help you check if the model's answers match expected results.

🎯 Goal: Create a LangChain evaluation pipeline that loads a set of questions and expected answers, configures a simple evaluation threshold, runs the evaluation by comparing model answers to expected answers, and finally outputs the evaluation results.

📋 What You'll Learn

Create a dictionary called test_data with three questions as keys and their expected answers as values.

Add a variable called accuracy_threshold set to 0.7 to configure the minimum acceptable accuracy.

Write a function called evaluate_model that takes test_data and returns the accuracy by comparing model answers to expected answers.

Add a final line that calls evaluate_model(test_data) and stores the result in a variable called evaluation_result.

💡 Why This Matters

🌍 Real World

Automated evaluation pipelines help developers quickly check if language models perform as expected on test questions without manual review.

💼 Career

Understanding how to build evaluation pipelines is useful for AI engineers and developers working with language models to ensure quality and reliability.

1

DATA SETUP: Create test data dictionary

Create a dictionary called test_data with these exact entries: 'What is the capital of France?': 'Paris', 'What color is the sky?': 'Blue', and 'How many legs does a spider have?': '8'.

LangChain

# Create the test_data dictionary with questions and expected answers
# Your code here

Need a hint?

Use curly braces {} to create a dictionary with the exact question-answer pairs.

2

CONFIGURATION: Set accuracy threshold

Add a variable called accuracy_threshold and set it to 0.7 to represent the minimum acceptable accuracy for the evaluation.

LangChain

test_data = {
    'What is the capital of France?': 'Paris',
    'What color is the sky?': 'Blue',
    'How many legs does a spider have?': '8'
}
# Set the accuracy_threshold variable to 0.7
# Your code here

Need a hint?

Just create a variable named accuracy_threshold and assign it the value 0.7.

3

CORE LOGIC: Write evaluation function

Write a function called evaluate_model that takes test_data as input. Inside, create a variable correct set to 0. Use a for loop with variables question and expected_answer to iterate over test_data.items(). For each question, simulate the model answer by setting model_answer = expected_answer. If model_answer equals expected_answer, increment correct by 1. Finally, return the accuracy as correct / len(test_data).

LangChain

test_data = {
    'What is the capital of France?': 'Paris',
    'What color is the sky?': 'Blue',
    'How many legs does a spider have?': '8'
}
accuracy_threshold = 0.7

# Define the evaluate_model function below
# Your code here

Need a hint?

Use a function with a for loop to count correct answers and calculate accuracy.

4

COMPLETION: Run evaluation and store result

Add a line that calls evaluate_model(test_data) and stores the result in a variable called evaluation_result.

LangChain

test_data = {
    'What is the capital of France?': 'Paris',
    'What color is the sky?': 'Blue',
    'How many legs does a spider have?': '8'
}
accuracy_threshold = 0.7

def evaluate_model(test_data):
    correct = 0
    for question, expected_answer in test_data.items():
        model_answer = expected_answer
        if model_answer == expected_answer:
            correct += 1
    return correct / len(test_data)

# Call evaluate_model with test_data and save to evaluation_result
# Your code here

Need a hint?

Just assign the function call result to evaluation_result.