LangSmith evaluators in LangChain - Mini Project: Build & Apply

Using LangSmith Evaluators in LangChain

📖 Scenario: You are building a simple LangChain application that uses LangSmith evaluators to check the quality of generated text. This helps you understand how well your language model is performing in a real-world setting.

🎯 Goal: Build a LangChain script that sets up a LangSmith evaluator, configures it with a threshold, evaluates a sample output, and completes the evaluation process.

📋 What You'll Learn

Create a dictionary called sample_data with keys input and output and exact values

Create a variable called threshold set to 0.7

Use the LangSmithEvaluator class to create an evaluator instance with the threshold

Call the evaluate method on the evaluator with sample_data and store the result in evaluation_result

💡 Why This Matters

🌍 Real World

Evaluators help developers check if language model outputs meet quality standards automatically.

💼 Career

Understanding how to use LangSmith evaluators is useful for AI developers and data scientists working with language models.

1

DATA SETUP: Create sample data dictionary

Create a dictionary called sample_data with these exact entries: 'input': 'What is the capital of France?' and 'output': 'Paris is the capital of France.'

LangChain

# Create the sample_data dictionary with input and output keys
# Your code here

Need a hint?

Use curly braces to create a dictionary with two keys: 'input' and 'output'.

2

CONFIGURATION: Set evaluation threshold

Create a variable called threshold and set it to the float value 0.7

LangChain

sample_data = {
    'input': 'What is the capital of France?',
    'output': 'Paris is the capital of France.'
}

# Set the threshold variable to 0.7
# Your code here

Need a hint?

Use a simple assignment to create the threshold variable.

3

CORE LOGIC: Create LangSmithEvaluator instance

Import LangSmithEvaluator from langchain.evaluation and create an instance called evaluator using the threshold variable

LangChain

sample_data = {
    'input': 'What is the capital of France?',
    'output': 'Paris is the capital of France.'
}

threshold = 0.7

# Import LangSmithEvaluator and create evaluator with threshold
# Your code here

Need a hint?

Use the import statement and pass the threshold variable when creating the evaluator.

4

COMPLETION: Evaluate sample data and store result

Call the evaluate method on evaluator with sample_data and assign the result to a variable called evaluation_result

LangChain

from langchain.evaluation import LangSmithEvaluator

sample_data = {
    'input': 'What is the capital of France?',
    'output': 'Paris is the capital of France.'
}

threshold = 0.7

evaluator = LangSmithEvaluator(threshold=threshold)

# Evaluate sample_data and store in evaluation_result
# Your code here

Need a hint?

Use dot notation to call the evaluate method and assign the result.