Custom evaluation metrics in LangChain - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ component_behavior

intermediate

What output does this custom metric function produce?

Consider this Python function used as a custom evaluation metric in Langchain:

def custom_metric(predictions, references):
    correct = sum(p == r for p, r in zip(predictions, references))
    total = len(references)
    return correct / total if total > 0 else 0

What is the output of custom_metric(['a', 'b', 'c'], ['a', 'x', 'c'])?

LangChain

def custom_metric(predictions, references):
    correct = sum(p == r for p, r in zip(predictions, references))
    total = len(references)
    return correct / total if total > 0 else 0

result = custom_metric(['a', 'b', 'c'], ['a', 'x', 'c'])

A 0

B 0.3333333333333333

C 0.6666666666666666

D 1.0

💡 Hint

Count how many predictions match the references exactly, then divide by total items.

The function counts matches: 'a'=='a' (true), 'b'=='x' (false), 'c'=='c' (true). So 2 matches out of 3 total, giving 2/3 = 0.666...

📝 Syntax

intermediate

Which option correctly defines a custom metric function in Langchain?

Which of the following Python functions correctly defines a custom evaluation metric that returns the ratio of matching items between predictions and references?

A

def metric(predictions, references):
    return sum(p == r for p, r in zip(predictions, references)) * len(references)

B

def metric(predictions, references):
    return sum(p == r for p in predictions for r in references) / len(references)

C

def metric(predictions, references):
    return sum(predictions == references) / len(references)

D

def metric(predictions, references):
    return sum(p == r for p, r in zip(predictions, references)) / len(references)

💡 Hint

Use zip to pair predictions and references correctly.

Option D correctly pairs each prediction with its reference and counts matches, then divides by total references.

Option D incorrectly uses nested loops causing overcounting.

Option D tries to compare lists directly, which is invalid.

Option D multiplies instead of dividing, giving wrong scale.

🔧 Debug

advanced

What error does this custom metric code raise?

Given this custom metric function:

def metric(predictions, references):
    return sum(p == r for p, r in zip(predictions, references)) / len(predictions)

What error will occur if predictions is an empty list and references is non-empty?

LangChain

def metric(predictions, references):
    return sum(p == r for p, r in zip(predictions, references)) / len(predictions)

result = metric([], ['a', 'b'])

A ZeroDivisionError

B IndexError

C TypeError

D No error, returns 0

💡 Hint

Check what happens when dividing by the length of an empty list.

Since len(predictions) is zero, dividing by zero causes a ZeroDivisionError.

🧠 Conceptual

advanced

Why use custom evaluation metrics in Langchain?

Which reason best explains why you might create a custom evaluation metric instead of using built-in ones in Langchain?

A To measure specific qualities of your model's output that built-in metrics don't capture

B Because built-in metrics are always inaccurate and unreliable

C Because Langchain requires custom metrics for all models

D To make the evaluation run faster by avoiding built-in functions

💡 Hint

Think about why general metrics might not fit every use case.

Custom metrics let you measure exactly what matters for your task, beyond generic built-in metrics.

Built-in metrics are not always inaccurate, and Langchain does not require custom metrics for all models.

❓ state_output

expert

What is the final value of score after running this custom metric?

Consider this code snippet used in Langchain to evaluate predictions:

class CustomMetric:
    def __init__(self):
        self.correct = 0
        self.total = 0
    def update(self, predictions, references):
        for p, r in zip(predictions, references):
            if p == r:
                self.correct += 1
            self.total += 1
    def compute(self):
        return self.correct / self.total if self.total > 0 else 0

metric = CustomMetric()
metric.update(['a', 'b'], ['a', 'x'])
metric.update(['c'], ['c'])
score = metric.compute()

What is the value of score?

LangChain

class CustomMetric:
    def __init__(self):
        self.correct = 0
        self.total = 0
    def update(self, predictions, references):
        for p, r in zip(predictions, references):
            if p == r:
                self.correct += 1
            self.total += 1
    def compute(self):
        return self.correct / self.total if self.total > 0 else 0

metric = CustomMetric()
metric.update(['a', 'b'], ['a', 'x'])
metric.update(['c'], ['c'])
score = metric.compute()

A 0.5

B 0.6666666666666666

C 1.0

D 0.0

💡 Hint

Count total matches and total items after both updates.

First update: matches 'a'=='a' (1), total 2.
Second update: matches 'c'=='c' (1), total 1.
Total correct = 2, total = 3, so score = 2/3 = 0.666...