Automated evaluation pipelines in LangChain - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ component_behavior

intermediate

What is the output of this LangChain evaluation pipeline snippet?

Consider this LangChain evaluation pipeline code that runs a simple evaluation on a list of inputs.

from langchain.evaluation import EvaluationChain
from langchain.schema import Document

def simple_eval_fn(doc: Document) -> bool:
    return "good" in doc.page_content

eval_chain = EvaluationChain.from_llm(llm=None, evaluation_fn=simple_eval_fn)
inputs = [Document(page_content="This is a good example."), Document(page_content="This is bad.")]
results = [eval_chain.evaluate(input_doc) for input_doc in inputs]
print(results)

What will be printed?

LangChain

from langchain.evaluation import EvaluationChain
from langchain.schema import Document

def simple_eval_fn(doc: Document) -> bool:
    return "good" in doc.page_content

eval_chain = EvaluationChain.from_llm(llm=None, evaluation_fn=simple_eval_fn)
inputs = [Document(page_content="This is a good example."), Document(page_content="This is bad.")]
results = [eval_chain.evaluate(input_doc) for input_doc in inputs]
print(results)

A TypeError: 'NoneType' object is not callable

B [False, True]

C ["good", "bad"]

D [True, False]

💡 Hint

Check what the evaluation function returns for each document.

The evaluation function returns True if the word "good" is in the document content. The first document contains "good", so True. The second does not, so False.

📝 Syntax

intermediate

Which option correctly creates a LangChain evaluation pipeline with a custom metric?

You want to create an evaluation pipeline that uses a custom metric function returning a float score. Which code snippet is syntactically correct?

A eval_chain = EvaluationChain.from_llm(llm=my_llm, evaluation_fn=lambda doc: 0.9)

B eval_chain = EvaluationChain.from_llm(llm=my_llm, evaluation_fn=lambda doc: return 0.9)

C eval_chain = EvaluationChain.from_llm(llm=my_llm, evaluation_fn=lambda doc: {0.9})

D eval_chain = EvaluationChain.from_llm(llm=my_llm, evaluation_fn=lambda doc: [0.9])

💡 Hint

Remember how lambda functions return values in Python.

Option A correctly uses a lambda that returns 0.9. Option A has a syntax error because 'return' cannot be used inside a lambda. Options C and D return a set and list, not a float.

🔧 Debug

advanced

Why does this LangChain evaluation pipeline raise an AttributeError?

Given this code snippet:

from langchain.evaluation import EvaluationChain

class MyEval:
    def __call__(self, doc):
        return len(doc.page_content)

eval_chain = EvaluationChain.from_llm(llm=None, evaluation_fn=MyEval())
result = eval_chain.evaluate("Test input")
print(result)

Why does it raise an AttributeError?

LangChain

from langchain.evaluation import EvaluationChain

class MyEval:
    def __call__(self, doc):
        return len(doc.page_content)

eval_chain = EvaluationChain.from_llm(llm=None, evaluation_fn=MyEval())
result = eval_chain.evaluate("Test input")
print(result)

A 'MyEval' object is not callable because __call__ is missing

B 'str' object has no attribute 'page_content' because evaluate was called with a string, not a Document

C TypeError because llm=None is invalid

D NameError because EvaluationChain is not imported

💡 Hint

Check the type of the argument passed to the evaluation function.

The evaluate method expects a Document object with a page_content attribute. Passing a string causes an AttributeError.

❓ state_output

advanced

What is the value of 'scores' after running this LangChain evaluation pipeline?

Consider this code:

from langchain.evaluation import EvaluationChain
from langchain.schema import Document

scores = []
def eval_fn(doc: Document) -> float:
    score = len(doc.page_content) / 10
    scores.append(score)
    return score

eval_chain = EvaluationChain.from_llm(llm=None, evaluation_fn=eval_fn)
inputs = [Document(page_content="Hello world!"), Document(page_content="LangChain")]
results = [eval_chain.evaluate(doc) for doc in inputs]

What is the final value of the list scores?

LangChain

from langchain.evaluation import EvaluationChain
from langchain.schema import Document

scores = []
def eval_fn(doc: Document) -> float:
    score = len(doc.page_content) / 10
    scores.append(score)
    return score

eval_chain = EvaluationChain.from_llm(llm=None, evaluation_fn=eval_fn)
inputs = [Document(page_content="Hello world!"), Document(page_content="LangChain")]
results = [eval_chain.evaluate(doc) for doc in inputs]

A [11, 9]

B [12, 9]

C [1.2, 0.9]

D [1.1, 0.9]

💡 Hint

Count the characters in each string and divide by 10.

"Hello world!" has 12 characters, so 12/10 = 1.2. "LangChain" has 9 characters, so 9/10 = 0.9.

🧠 Conceptual

expert

Which option best describes the role of 'evaluation_fn' in LangChain's EvaluationChain?

In LangChain's EvaluationChain, what is the primary purpose of the evaluation_fn parameter?

A It defines a function that takes a Document and returns a metric or boolean indicating evaluation results

B It specifies the language model used to generate text outputs

C It configures the logging level for the evaluation process

D It initializes the database connection for storing evaluation data

💡 Hint

Think about what evaluation means in this context.

The evaluation_fn is a function that receives a Document and returns a score or boolean to indicate how well the document meets criteria.