Creating evaluation datasets in LangChain - Try It Yourself

Creating Evaluation Datasets with Langchain

📖 Scenario: You are building a simple evaluation dataset for a language model. This dataset will contain questions and their correct answers. You want to organize this data so you can later use it to test how well your model performs.

🎯 Goal: Create a dictionary with questions and answers, set a threshold for evaluation, filter the dataset based on the threshold, and finalize the dataset for use in Langchain evaluation.

📋 What You'll Learn

Create a dictionary called qa_pairs with 3 exact question-answer pairs

Create a variable called min_score set to 0.7

Use a dictionary comprehension to create filtered_qa with only pairs having scores above min_score

Add a final dictionary called evaluation_dataset that includes filtered_qa and a description string

💡 Why This Matters

🌍 Real World

Evaluation datasets help test how well language models answer questions correctly before using them in real applications.

💼 Career

Creating and managing evaluation datasets is a key skill for AI developers and data scientists working with language models.

1

Create the initial question-answer dictionary

Create a dictionary called qa_pairs with these exact entries: 'What is AI?': 'Artificial Intelligence', 'What is ML?': 'Machine Learning', 'What is NLP?': 'Natural Language Processing'.

LangChain

# Create the qa_pairs dictionary with 3 question-answer pairs
# Your code here

Hint

Use curly braces {} to create a dictionary with keys as questions and values as answers.

2

Add a minimum score threshold

Create a variable called min_score and set it to 0.7.

LangChain

qa_pairs = {
    'What is AI?': 'Artificial Intelligence',
    'What is ML?': 'Machine Learning',
    'What is NLP?': 'Natural Language Processing'
}
# Set the minimum score threshold
# Your code here

Hint

Just assign the number 0.7 to the variable min_score.

3

Filter the dataset based on scores

Create a dictionary called filtered_qa using dictionary comprehension. Include only the pairs from qa_pairs where the score is above min_score. Use this exact scores dictionary: scores = {'What is AI?': 0.9, 'What is ML?': 0.65, 'What is NLP?': 0.8}.

LangChain

qa_pairs = {
    'What is AI?': 'Artificial Intelligence',
    'What is ML?': 'Machine Learning',
    'What is NLP?': 'Natural Language Processing'
}
min_score = 0.7
scores = {'What is AI?': 0.9, 'What is ML?': 0.65, 'What is NLP?': 0.8}
# Use dictionary comprehension to filter qa_pairs by min_score
# Your code here

Hint

Use {key: value for key, value in dict.items() if condition} to filter.

4

Create the final evaluation dataset dictionary

Create a dictionary called evaluation_dataset with two keys: 'data' set to filtered_qa and 'description' set to the string 'Filtered QA pairs for evaluation'.

LangChain

qa_pairs = {
    'What is AI?': 'Artificial Intelligence',
    'What is ML?': 'Machine Learning',
    'What is NLP?': 'Natural Language Processing'
}
min_score = 0.7
scores = {'What is AI?': 0.9, 'What is ML?': 0.65, 'What is NLP?': 0.8}
filtered_qa = {q: a for q, a in qa_pairs.items() if scores[q] > min_score}
# Create evaluation_dataset dictionary with data and description
# Your code here

Hint

Create a dictionary with keys 'data' and 'description' and assign the correct values.

Practice

1. What is the main purpose of creating evaluation datasets in LangChain?

easy

A. To speed up the language model's response time

B. To train the language model with more data

C. To test how well the language model answers specific questions

D. To store user conversations permanently

Solution

1. Step 1: Understand evaluation datasets

   Evaluation datasets contain example questions and expected answers to check model accuracy.

2. Step 2: Identify the purpose in LangChain context

   They are used to test how well the model answers, not for training or storage.

3. Final Answer:

   To test how well the language model answers specific questions -> Option C

4. Quick Check:

   Evaluation datasets = test model accuracy [OK]

Hint: Evaluation datasets check model answers, not train it [OK]

Common Mistakes:

• Confusing evaluation datasets with training data
• Thinking evaluation datasets speed up the model
• Assuming evaluation datasets store user data

2. Which of the following is the correct way to create an evaluation example in LangChain?

easy

A. example = ("What is AI?", "Artificial Intelligence")

B. example = "What is AI? -> Artificial Intelligence"

C. example = ["What is AI?", "Artificial Intelligence"]

D. example = {"query": "What is AI?", "expected_answer": "Artificial Intelligence"}

Solution

1. Step 1: Recall LangChain evaluation example format

   Evaluation examples are dictionaries with keys like 'query' and 'expected_answer'.

2. Step 2: Match the correct syntax

   example = {"query": "What is AI?", "expected_answer": "Artificial Intelligence"} uses a dictionary with proper keys, others use tuples, lists, or strings incorrectly.

3. Final Answer:

   example = {"query": "What is AI?", "expected_answer": "Artificial Intelligence"} -> Option D

4. Quick Check:

   Evaluation example = dictionary with keys [OK]

Hint: Use dictionary with 'query' and 'expected_answer' keys [OK]

Common Mistakes:

• Using tuples or lists instead of dictionaries
• Not using correct keys 'query' and 'expected_answer'
• Using plain strings without structure

3. Given the following code snippet, what will be the output?

from langchain.evaluation.qa import QAEvalChain
examples = [{"query": "Capital of France?", "expected_answer": "Paris"}]
chain = QAEvalChain.from_llm(llm=None)
results = chain.evaluate(examples)
print(results)

medium

A. TypeError because llm=None is invalid

B. SyntaxError due to missing import

C. Empty list [] because no LLM provided

D. [{'query': 'Capital of France?', 'expected_answer': 'Paris', 'result': 'correct'}]

Solution

1. Step 1: Analyze the QAEvalChain initialization

   The method from_llm requires a valid language model instance, not None.

2. Step 2: Predict the error from invalid llm argument

   Passing None will cause a TypeError or similar because the chain cannot run without a model.

3. Final Answer:

   TypeError because llm=None is invalid -> Option A

4. Quick Check:

   Invalid llm argument = TypeError [OK]

Hint: QAEvalChain needs a valid LLM, None causes error [OK]

Common Mistakes:

• Assuming None is a valid LLM
• Expecting output without running the model
• Ignoring required imports or parameters

4. You wrote this code to create evaluation examples but get an error:

examples = [{"query": "Who wrote Hamlet?", "answer": "Shakespeare"}]
chain = QAEvalChain.from_llm(llm=some_llm)
results = chain.evaluate(examples)
print(results)

What is the likely cause of the error?

medium

A. The variable some_llm is not defined

B. The key 'answer' should be 'expected_answer' in the example dictionary

C. QAEvalChain does not have an evaluate method

D. The examples list should be empty

Solution

1. Step 1: Check example dictionary keys

   LangChain expects 'expected_answer' key, not 'answer', for evaluation examples.

2. Step 2: Identify mismatch causing error

   Using 'answer' instead of 'expected_answer' causes the chain to fail reading expected answers.

3. Final Answer:

   The key 'answer' should be 'expected_answer' in the example dictionary -> Option B

4. Quick Check:

   Correct key name = 'expected_answer' [OK]

Hint: Use 'expected_answer' key, not 'answer' in examples [OK]

Common Mistakes:

• Using wrong key names in example dictionaries
• Assuming method names without checking docs
• Ignoring variable definitions

5. You want to create an evaluation dataset with multiple examples and run QAEvalChain to check model accuracy. Which approach correctly prepares and evaluates the dataset?

hard

A. Prepare a list of dictionaries with 'query' and 'expected_answer', then call chain.evaluate(examples)

B. Prepare a list of tuples (query, expected_answer), then call chain.run(examples)

C. Prepare a dictionary with queries as keys and answers as values, then call chain.evaluate(examples)

D. Prepare a list of strings with 'query: answer' format, then call chain.run(examples)

Solution

1. Step 1: Format evaluation dataset correctly

   LangChain expects a list of dictionaries with keys 'query' and 'expected_answer' for evaluation.

2. Step 2: Use the correct method to evaluate

   The QAEvalChain uses the evaluate() method to process multiple examples at once.

3. Final Answer:

   Prepare a list of dictionaries with 'query' and 'expected_answer', then call chain.evaluate(examples) -> Option A

4. Quick Check:

   List of dicts + evaluate() = correct approach [OK]

Hint: Use list of dicts with evaluate() method for multiple examples [OK]

Common Mistakes:

• Using tuples or dicts with wrong structure
• Calling run() instead of evaluate() for batch evaluation
• Passing strings instead of structured data