Creating evaluation datasets in LangChain - Try It Yourself

Creating Evaluation Datasets with Langchain

📖 Scenario: You are building a simple evaluation dataset for a language model. This dataset will contain questions and their correct answers. You want to organize this data so you can later use it to test how well your model performs.

🎯 Goal: Create a dictionary with questions and answers, set a threshold for evaluation, filter the dataset based on the threshold, and finalize the dataset for use in Langchain evaluation.

📋 What You'll Learn

Create a dictionary called qa_pairs with 3 exact question-answer pairs

Create a variable called min_score set to 0.7

Use a dictionary comprehension to create filtered_qa with only pairs having scores above min_score

Add a final dictionary called evaluation_dataset that includes filtered_qa and a description string

💡 Why This Matters

🌍 Real World

Evaluation datasets help test how well language models answer questions correctly before using them in real applications.

💼 Career

Creating and managing evaluation datasets is a key skill for AI developers and data scientists working with language models.

1

Create the initial question-answer dictionary

Create a dictionary called qa_pairs with these exact entries: 'What is AI?': 'Artificial Intelligence', 'What is ML?': 'Machine Learning', 'What is NLP?': 'Natural Language Processing'.

LangChain

# Create the qa_pairs dictionary with 3 question-answer pairs
# Your code here

Need a hint?

Use curly braces {} to create a dictionary with keys as questions and values as answers.

2

Add a minimum score threshold

Create a variable called min_score and set it to 0.7.

LangChain

qa_pairs = {
    'What is AI?': 'Artificial Intelligence',
    'What is ML?': 'Machine Learning',
    'What is NLP?': 'Natural Language Processing'
}
# Set the minimum score threshold
# Your code here

Need a hint?

Just assign the number 0.7 to the variable min_score.

3

Filter the dataset based on scores

Create a dictionary called filtered_qa using dictionary comprehension. Include only the pairs from qa_pairs where the score is above min_score. Use this exact scores dictionary: scores = {'What is AI?': 0.9, 'What is ML?': 0.65, 'What is NLP?': 0.8}.

LangChain

qa_pairs = {
    'What is AI?': 'Artificial Intelligence',
    'What is ML?': 'Machine Learning',
    'What is NLP?': 'Natural Language Processing'
}
min_score = 0.7
scores = {'What is AI?': 0.9, 'What is ML?': 0.65, 'What is NLP?': 0.8}
# Use dictionary comprehension to filter qa_pairs by min_score
# Your code here

Need a hint?

Use {key: value for key, value in dict.items() if condition} to filter.

4

Create the final evaluation dataset dictionary

Create a dictionary called evaluation_dataset with two keys: 'data' set to filtered_qa and 'description' set to the string 'Filtered QA pairs for evaluation'.

LangChain

qa_pairs = {
    'What is AI?': 'Artificial Intelligence',
    'What is ML?': 'Machine Learning',
    'What is NLP?': 'Natural Language Processing'
}
min_score = 0.7
scores = {'What is AI?': 0.9, 'What is ML?': 0.65, 'What is NLP?': 0.8}
filtered_qa = {q: a for q, a in qa_pairs.items() if scores[q] > min_score}
# Create evaluation_dataset dictionary with data and description
# Your code here

Need a hint?

Create a dictionary with keys 'data' and 'description' and assign the correct values.