Why evaluation prevents production failures in LangChain - Test Your Understanding

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create an evaluation that checks if the output matches the expected answer.

LangChain

from langchain.evaluation import StringEvaluator

evaluator = StringEvaluator()
result = evaluator.evaluate(output=[1], reference="Hello World")

Drag options to blanks, or click blank then click option'

Aoutput_text

Bresponse

Coutput

Dinput_text

Attempts:

3 left

💡 Hint

Common Mistakes

Using the input variable instead of the output.

Passing a variable that is not defined.

The evaluate method requires the actual output to compare with the reference. Here, output is the correct variable holding the generated result.

2fill in blank

medium

Complete the code to run an evaluation on a chain's output using LangChain's evaluation tools.

LangChain

from langchain.chains import LLMChain
from langchain.evaluation import StringEvaluator

chain = LLMChain(llm=llm, prompt=prompt)
output = chain.run("What is 2 + 2?")
evaluator = StringEvaluator()
score = evaluator.evaluate(output=[1], reference="4")

Drag options to blanks, or click blank then click option'

Aoutput

Bresult

Cresponse

Danswer

Attempts:

3 left

💡 Hint

Common Mistakes

Passing an undefined variable to the evaluator.

Using the input prompt instead of the output.

The variable output holds the chain's generated answer, which is what the evaluator needs to check against the reference.

3fill in blank

hard

Fix the error in the evaluation code by completing the blank with the correct method to get the chain's output.

LangChain

output = chain.[1]("Calculate 5 times 3")
score = evaluator.evaluate(output=output, reference="15")

Drag options to blanks, or click blank then click option'

Arun

Bexecute

Ccall

Dprocess

Attempts:

3 left

💡 Hint

Common Mistakes

Using non-existent methods like execute or process.

Confusing call with run.

The run method is the correct way to execute the chain and get its output as a string.

4fill in blank

hard

Fill both blanks to create a dictionary comprehension that evaluates outputs only if the score is above a threshold.

LangChain

results = {output: evaluator.evaluate(output=output, reference=ref) for output, ref in outputs.items() if evaluator.evaluate(output=output, reference=ref) [1] [2]

Drag options to blanks, or click blank then click option'

A>

B==

C0.8

D0.5

Attempts:

3 left

💡 Hint

Common Mistakes

Using equality instead of greater than.

Choosing a threshold too low to be meaningful.

The comprehension filters results where the evaluation score is greater than 0.8, meaning only good matches are included.

5fill in blank

hard

Fill all three blanks to define a function that evaluates a list of outputs and returns those with scores above a threshold.

LangChain

def filter_good_outputs(outputs, evaluator, threshold):
    return {output: score for output in outputs if (score := evaluator.evaluate(output=output, reference=outputs[output])) [1] threshold and score [2] 1 and output [3] outputs}

Drag options to blanks, or click blank then click option'

A>

B<=

Cin

Dnot in

Attempts:

3 left

💡 Hint

Common Mistakes

Using wrong comparison operators that invert logic.

Checking if output is not in outputs instead of in outputs.

The function keeps outputs with scores greater than the threshold and less than or equal to 1, ensuring valid scores. It also checks that the output is a key in the outputs dictionary.