Creating evaluation datasets in LangChain - Interactive Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create an empty list for evaluation examples.

LangChain

evaluation_examples = [1]

Drag options to blanks, or click blank then click option'

A()

B{}

C[]

Dset()

Attempts:

3 left

💡 Hint

Common Mistakes

Using curly braces {} creates a dictionary, not a list.

Using parentheses () creates a tuple, which is immutable.

We use square brackets [] to create an empty list in Python, which is suitable for storing evaluation examples.

2fill in blank

medium

Complete the code to add a dictionary with input and output to the evaluation examples list.

LangChain

evaluation_examples.append([1])

Drag options to blanks, or click blank then click option'

A{'input': 'Hello', 'output': 'Hi there!'}

B('input', 'Hello', 'output', 'Hi there!')

C['input', 'Hello', 'output', 'Hi there!']

D'input: Hello, output: Hi there!'

Attempts:

3 left

💡 Hint

Common Mistakes

Using a list or tuple instead of a dictionary.

Adding a plain string instead of a dictionary.

Evaluation examples are stored as dictionaries with keys like input and output. So we add a dictionary to the list.

3fill in blank

hard

Fix the error in the code to create a list of evaluation examples with two entries.

LangChain

evaluation_examples = [[1]]

Drag options to blanks, or click blank then click option'

A{'input': 'Hi', 'output': 'Hello'} {'input': 'Bye', 'output': 'Goodbye'}

B{'input': 'Hi', 'output': 'Hello'}, {'input': 'Bye', 'output': 'Goodbye'}

C[{'input': 'Hi', 'output': 'Hello'}, {'input': 'Bye', 'output': 'Goodbye'}]

D({'input': 'Hi', 'output': 'Hello'}, {'input': 'Bye', 'output': 'Goodbye'})

Attempts:

3 left

💡 Hint

Common Mistakes

Missing commas between dictionaries.

Using extra brackets or parentheses inside the list.

Inside the list brackets, multiple dictionaries are separated by commas without extra brackets.

4fill in blank

hard

Complete the code to create a dictionary comprehension that filters evaluation examples with input length greater than 3.

LangChain

filtered_examples = {word:len(word) for word in evaluation_examples if len(word['input']) [1] 3}

Drag options to blanks, or click blank then click option'

A:

B>

C<

D=

Attempts:

3 left

💡 Hint

Common Mistakes

Using = instead of : in dictionary comprehension.

Using wrong comparison operators like < or =.

In dictionary comprehensions, : separates key and value. The condition checks if input length is greater than 3 using >.

5fill in blank

hard

Complete the code to create a list comprehension extracting outputs for inputs containing 'test'.

LangChain

test_outputs = [example['output'] for example in evaluation_examples if 'test' [1] example['input']]

Drag options to blanks, or click blank then click option'

A[

Bin

C]

Dnot in

Attempts:

3 left

💡 Hint

Common Mistakes

Forgetting to open or close the list brackets.

Using not in instead of in for the condition.

The list comprehension starts with [ and ends with ]. The condition uses in to check if 'test' is in the input string.

Practice

1. What is the main purpose of creating evaluation datasets in LangChain?

easy

A. To speed up the language model's response time

B. To train the language model with more data

C. To test how well the language model answers specific questions

D. To store user conversations permanently

Solution

1. Step 1: Understand evaluation datasets

   Evaluation datasets contain example questions and expected answers to check model accuracy.

2. Step 2: Identify the purpose in LangChain context

   They are used to test how well the model answers, not for training or storage.

3. Final Answer:

   To test how well the language model answers specific questions -> Option C

4. Quick Check:

   Evaluation datasets = test model accuracy [OK]

Hint: Evaluation datasets check model answers, not train it [OK]

Common Mistakes:

• Confusing evaluation datasets with training data
• Thinking evaluation datasets speed up the model
• Assuming evaluation datasets store user data

2. Which of the following is the correct way to create an evaluation example in LangChain?

easy

A. example = ("What is AI?", "Artificial Intelligence")

B. example = "What is AI? -> Artificial Intelligence"

C. example = ["What is AI?", "Artificial Intelligence"]

D. example = {"query": "What is AI?", "expected_answer": "Artificial Intelligence"}

Solution

1. Step 1: Recall LangChain evaluation example format

   Evaluation examples are dictionaries with keys like 'query' and 'expected_answer'.

2. Step 2: Match the correct syntax

   example = {"query": "What is AI?", "expected_answer": "Artificial Intelligence"} uses a dictionary with proper keys, others use tuples, lists, or strings incorrectly.

3. Final Answer:

   example = {"query": "What is AI?", "expected_answer": "Artificial Intelligence"} -> Option D

4. Quick Check:

   Evaluation example = dictionary with keys [OK]

Hint: Use dictionary with 'query' and 'expected_answer' keys [OK]

Common Mistakes:

• Using tuples or lists instead of dictionaries
• Not using correct keys 'query' and 'expected_answer'
• Using plain strings without structure

3. Given the following code snippet, what will be the output?

from langchain.evaluation.qa import QAEvalChain
examples = [{"query": "Capital of France?", "expected_answer": "Paris"}]
chain = QAEvalChain.from_llm(llm=None)
results = chain.evaluate(examples)
print(results)

medium

A. TypeError because llm=None is invalid

B. SyntaxError due to missing import

C. Empty list [] because no LLM provided

D. [{'query': 'Capital of France?', 'expected_answer': 'Paris', 'result': 'correct'}]

Solution

1. Step 1: Analyze the QAEvalChain initialization

   The method from_llm requires a valid language model instance, not None.

2. Step 2: Predict the error from invalid llm argument

   Passing None will cause a TypeError or similar because the chain cannot run without a model.

3. Final Answer:

   TypeError because llm=None is invalid -> Option A

4. Quick Check:

   Invalid llm argument = TypeError [OK]

Hint: QAEvalChain needs a valid LLM, None causes error [OK]

Common Mistakes:

• Assuming None is a valid LLM
• Expecting output without running the model
• Ignoring required imports or parameters

4. You wrote this code to create evaluation examples but get an error:

examples = [{"query": "Who wrote Hamlet?", "answer": "Shakespeare"}]
chain = QAEvalChain.from_llm(llm=some_llm)
results = chain.evaluate(examples)
print(results)

What is the likely cause of the error?

medium

A. The variable some_llm is not defined

B. The key 'answer' should be 'expected_answer' in the example dictionary

C. QAEvalChain does not have an evaluate method

D. The examples list should be empty

Solution

1. Step 1: Check example dictionary keys

   LangChain expects 'expected_answer' key, not 'answer', for evaluation examples.

2. Step 2: Identify mismatch causing error

   Using 'answer' instead of 'expected_answer' causes the chain to fail reading expected answers.

3. Final Answer:

   The key 'answer' should be 'expected_answer' in the example dictionary -> Option B

4. Quick Check:

   Correct key name = 'expected_answer' [OK]

Hint: Use 'expected_answer' key, not 'answer' in examples [OK]

Common Mistakes:

• Using wrong key names in example dictionaries
• Assuming method names without checking docs
• Ignoring variable definitions

5. You want to create an evaluation dataset with multiple examples and run QAEvalChain to check model accuracy. Which approach correctly prepares and evaluates the dataset?

hard

A. Prepare a list of dictionaries with 'query' and 'expected_answer', then call chain.evaluate(examples)

B. Prepare a list of tuples (query, expected_answer), then call chain.run(examples)

C. Prepare a dictionary with queries as keys and answers as values, then call chain.evaluate(examples)

D. Prepare a list of strings with 'query: answer' format, then call chain.run(examples)

Solution

1. Step 1: Format evaluation dataset correctly

   LangChain expects a list of dictionaries with keys 'query' and 'expected_answer' for evaluation.

2. Step 2: Use the correct method to evaluate

   The QAEvalChain uses the evaluate() method to process multiple examples at once.

3. Final Answer:

   Prepare a list of dictionaries with 'query' and 'expected_answer', then call chain.evaluate(examples) -> Option A

4. Quick Check:

   List of dicts + evaluate() = correct approach [OK]

Hint: Use list of dicts with evaluate() method for multiple examples [OK]

Common Mistakes:

• Using tuples or dicts with wrong structure
• Calling run() instead of evaluate() for batch evaluation
• Passing strings instead of structured data