Automated evaluation pipelines in LangChain - Mini Project: Build & Apply

Automated Evaluation Pipelines with LangChain

📖 Scenario: You are building a simple automated evaluation pipeline using LangChain to test how well a language model answers questions. This pipeline will help you check if the model's answers match expected results.

🎯 Goal: Create a LangChain evaluation pipeline that loads a set of questions and expected answers, configures a simple evaluation threshold, runs the evaluation by comparing model answers to expected answers, and finally outputs the evaluation results.

📋 What You'll Learn

Create a dictionary called test_data with three questions as keys and their expected answers as values.

Add a variable called accuracy_threshold set to 0.7 to configure the minimum acceptable accuracy.

Write a function called evaluate_model that takes test_data and returns the accuracy by comparing model answers to expected answers.

Add a final line that calls evaluate_model(test_data) and stores the result in a variable called evaluation_result.

💡 Why This Matters

🌍 Real World

Automated evaluation pipelines help developers quickly check if language models perform as expected on test questions without manual review.

💼 Career

Understanding how to build evaluation pipelines is useful for AI engineers and developers working with language models to ensure quality and reliability.

1

DATA SETUP: Create test data dictionary

Create a dictionary called test_data with these exact entries: 'What is the capital of France?': 'Paris', 'What color is the sky?': 'Blue', and 'How many legs does a spider have?': '8'.

LangChain

# Create the test_data dictionary with questions and expected answers
# Your code here

Hint

Use curly braces {} to create a dictionary with the exact question-answer pairs.

2

CONFIGURATION: Set accuracy threshold

Add a variable called accuracy_threshold and set it to 0.7 to represent the minimum acceptable accuracy for the evaluation.

LangChain

test_data = {
    'What is the capital of France?': 'Paris',
    'What color is the sky?': 'Blue',
    'How many legs does a spider have?': '8'
}
# Set the accuracy_threshold variable to 0.7
# Your code here

Hint

Just create a variable named accuracy_threshold and assign it the value 0.7.

3

CORE LOGIC: Write evaluation function

Write a function called evaluate_model that takes test_data as input. Inside, create a variable correct set to 0. Use a for loop with variables question and expected_answer to iterate over test_data.items(). For each question, simulate the model answer by setting model_answer = expected_answer. If model_answer equals expected_answer, increment correct by 1. Finally, return the accuracy as correct / len(test_data).

LangChain

test_data = {
    'What is the capital of France?': 'Paris',
    'What color is the sky?': 'Blue',
    'How many legs does a spider have?': '8'
}
accuracy_threshold = 0.7

# Define the evaluate_model function below
# Your code here

Hint

Use a function with a for loop to count correct answers and calculate accuracy.

4

COMPLETION: Run evaluation and store result

Add a line that calls evaluate_model(test_data) and stores the result in a variable called evaluation_result.

LangChain

test_data = {
    'What is the capital of France?': 'Paris',
    'What color is the sky?': 'Blue',
    'How many legs does a spider have?': '8'
}
accuracy_threshold = 0.7

def evaluate_model(test_data):
    correct = 0
    for question, expected_answer in test_data.items():
        model_answer = expected_answer
        if model_answer == expected_answer:
            correct += 1
    return correct / len(test_data)

# Call evaluate_model with test_data and save to evaluation_result
# Your code here

Hint

Just assign the function call result to evaluation_result.

Practice

1. What is the main purpose of an automated evaluation pipeline in Langchain?

easy

A. To quickly test language model outputs against expected answers

B. To train new language models from scratch

C. To manually review each model output for quality

D. To deploy language models to production servers

Solution

1. Step 1: Understand the role of evaluation pipelines

   Evaluation pipelines automatically compare model outputs to expected answers to check correctness.

2. Step 2: Identify the main benefit

   This automation speeds up testing and helps catch errors early without manual review.

3. Final Answer:

   To quickly test language model outputs against expected answers -> Option A

4. Quick Check:

   Automated testing = Quick evaluation [OK]

Hint: Evaluation pipelines compare outputs to expected answers fast [OK]

Common Mistakes:

• Confusing evaluation with training
• Thinking evaluation is manual
• Assuming deployment is part of evaluation

2. Which of the following is the correct way to create an evaluation pipeline in Langchain?

easy

A. pipeline = EvaluationPipeline(inputs, model, expected_outputs)

B. pipeline = EvaluationPipeline(model, inputs, expected_outputs)

C. pipeline = EvaluationPipeline(expected_outputs, inputs, model)

D. pipeline = EvaluationPipeline(inputs, expected_outputs, model)

Solution

1. Step 1: Recall the order of parameters

   The EvaluationPipeline constructor expects inputs first, then the model, then expected outputs.

2. Step 2: Match the correct parameter order

   pipeline = EvaluationPipeline(inputs, model, expected_outputs) matches this order exactly, others mix the sequence causing errors.

3. Final Answer:

   pipeline = EvaluationPipeline(inputs, model, expected_outputs) -> Option A

4. Quick Check:

   Inputs, model, expected outputs order [OK]

Hint: Remember: inputs first, then model, then expected outputs [OK]

Common Mistakes:

• Swapping model and inputs order
• Putting expected outputs before inputs
• Using wrong parameter sequence causing errors

3. Given this code snippet, what will be the output of results?

inputs = ["Hello", "World"]
model = lambda x: x.lower()
expected = ["hello", "world"]
pipeline = EvaluationPipeline(inputs, model, expected)
results = pipeline.run()

medium

A. [True, False]

B. [False, False]

C. [True, True]

D. RuntimeError

Solution

1. Step 1: Understand the model function

   The model converts each input string to lowercase, so "Hello" -> "hello" and "World" -> "world".

2. Step 2: Compare model outputs to expected

   Both outputs match the expected list exactly, so evaluation returns True for both.

3. Final Answer:

   [True, True] -> Option C

4. Quick Check:

   Lowercase matches expected = True [OK]

Hint: Check if model output matches expected exactly [OK]

Common Mistakes:

• Assuming case does not matter
• Expecting runtime error from lambda
• Mixing up True and False results

4. You wrote this evaluation pipeline but it raises an error:

inputs = ["Test"]
model = "not a function"
expected = ["test"]
pipeline = EvaluationPipeline(inputs, model, expected)
pipeline.run()

What is the likely cause?

medium

A. Inputs list cannot have only one item

B. Model must be a callable function, not a string

C. Expected outputs must be integers

D. EvaluationPipeline requires three arguments, but only two were given

Solution

1. Step 1: Check the model parameter type

   The model should be a function that processes inputs, but here it is a string, which is not callable.

2. Step 2: Understand the error cause

   Calling pipeline.run() tries to call the model on inputs, causing a TypeError because strings can't be called like functions.

3. Final Answer:

   Model must be a callable function, not a string -> Option B

4. Quick Check:

   Model callable required, string given [OK]

Hint: Model must be a function, not a string [OK]

Common Mistakes:

• Thinking inputs size causes error
• Expecting output type to be integer
• Miscounting constructor arguments

5. You want to evaluate a language model that sometimes returns empty strings for some inputs. How should you modify your automated evaluation pipeline to handle this edge case correctly?

hard

A. Replace empty string outputs with None before evaluation

B. Treat empty string outputs as incorrect regardless of expected answer

C. Ignore inputs that produce empty strings in the evaluation

D. Filter out empty string outputs before comparing to expected answers

Solution

1. Step 1: Identify the problem with empty strings

   Empty string outputs can cause false negatives if compared directly to expected answers.

2. Step 2: Implement filtering before comparison

   Filtering out empty strings ensures only meaningful outputs are evaluated, avoiding misleading failures.

3. Step 3: Avoid ignoring inputs or forcing None

   Ignoring inputs or replacing outputs can hide real issues or cause errors in evaluation.

4. Final Answer:

   Filter out empty string outputs before comparing to expected answers -> Option D

5. Quick Check:

   Filter empty outputs to avoid false errors [OK]

Hint: Filter empty outputs before evaluation to avoid false failures [OK]

Common Mistakes:

• Ignoring inputs with empty outputs
• Replacing empty strings with None causing errors
• Counting empty strings as always wrong