Custom evaluation metrics in LangChain - Mini Project: Build & Apply

Custom Evaluation Metrics with Langchain

📖 Scenario: You are building a language model evaluation tool using Langchain. You want to create a custom metric to measure how well the model's answers match expected answers.

🎯 Goal: Build a simple custom evaluation metric function and integrate it with Langchain's evaluation framework.

📋 What You'll Learn

Create a list of model answers and expected answers

Define a threshold for exact match score

Write a function to calculate exact match accuracy

Use the function as a custom metric in Langchain evaluation

💡 Why This Matters

🌍 Real World

Custom evaluation metrics help you measure how well AI models perform on your specific tasks, beyond generic scores.

💼 Career

Knowing how to create and use custom metrics is valuable for AI engineers and data scientists working on model evaluation and improvement.

1

Data Setup: Create model and expected answers

Create a list called model_answers with these exact strings: 'Paris', 'Berlin', 'Tokyo'. Also create a list called expected_answers with these exact strings: 'Paris', 'Berlin', 'Kyoto'.

LangChain

# Your code here

Need a hint?

Use Python lists with exact string values as shown.

2

Configuration: Define exact match threshold

Create a variable called exact_match_threshold and set it to 1.0 to represent a perfect match score.

LangChain

model_answers = ['Paris', 'Berlin', 'Tokyo']
expected_answers = ['Paris', 'Berlin', 'Kyoto']
# Your code here

Need a hint?

Use a float value 1.0 to represent exact match threshold.

3

Core Logic: Write exact match accuracy function

Define a function called exact_match_accuracy that takes two lists: predictions and references. It should return the fraction of items where prediction equals reference exactly.

LangChain

model_answers = ['Paris', 'Berlin', 'Tokyo']
expected_answers = ['Paris', 'Berlin', 'Kyoto']
exact_match_threshold = 1.0
# Define exact_match_accuracy function here

Need a hint?

Use zip to pair predictions and references, then count exact matches.

4

Completion: Use the custom metric in Langchain evaluation

Import EvaluationChain from langchain.evaluation. Create an eval_chain instance using EvaluationChain.from_llm with a dummy llm=None and pass exact_match_accuracy as the metric argument.

LangChain

model_answers = ['Paris', 'Berlin', 'Tokyo']
expected_answers = ['Paris', 'Berlin', 'Kyoto']
exact_match_threshold = 1.0

def exact_match_accuracy(predictions, references):
    matches = sum(p == r for p, r in zip(predictions, references))
    return matches / len(references)

# Import EvaluationChain and create eval_chain with exact_match_accuracy metric

Need a hint?

Use from langchain.evaluation import EvaluationChain and pass your function as metric.