LangSmith evaluators in LangChain - Step-by-Step Execution

Concept Flow - LangSmith evaluators

Input: Model Output + Reference

↓

Evaluator Receives Data

↓

Run Evaluation Logic

↓

Generate Score or Feedback

↓

Return Evaluation Result

The evaluator takes the model output and reference, runs evaluation logic, and returns a score or feedback.

Execution Sample

LangChain

from langchain.evaluation import StringEvaluator

eval = StringEvaluator()
result = eval.evaluate_string(
    prediction="Hello world",
    reference="Hello world!"
)

This code creates a StringEvaluator and evaluates a prediction against a reference string.

Execution Table

Step	Action	Input	Evaluation Logic	Output
1	Create StringEvaluator instance	None	Initialize evaluator	Evaluator ready
2	Call evaluate_string	prediction='Hello world', reference='Hello world!'	Compare strings with tolerance	Score calculated
3	Return evaluation result	Score calculated	Format result	{"score": 0.9, "feedback": "Close match"}
4	End	Evaluation complete	No further action	Evaluation finished

💡 Evaluation completes after returning the score and feedback.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
eval	None	StringEvaluator instance	StringEvaluator instance	StringEvaluator instance	StringEvaluator instance
prediction	None	None	"Hello world"	"Hello world"	"Hello world"
reference	None	None	"Hello world!"	"Hello world!"	"Hello world!"
result	None	None	None	{"score": 0.9, "feedback": "Close match"}	{"score": 0.9, "feedback": "Close match"}

Key Moments - 2 Insights

Why does the evaluator return a score less than 1 even though the prediction looks very similar to the reference?

Because the evaluation logic compares strings exactly or with some tolerance, the missing exclamation mark causes a slight difference, resulting in a score less than 1 as shown in step 2 and 3 of the execution_table.

What happens if the prediction or reference is None or empty?

The evaluator will still run but may return a low or zero score because the comparison logic expects strings. This is implied in the input column of step 2 where valid strings are required.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output after step 3?

A {"score": 0.9, "feedback": "Close match"}

B Evaluator ready

C Score calculated

D Evaluation finished

💡 Hint

Check the Output column in row with Step 3.

At which step does the evaluator compare the prediction and reference strings?

A Step 1

B Step 3

C Step 2

D Step 4

💡 Hint

Look at the Evaluation Logic column to find where comparison happens.

If the prediction was exactly the same as the reference, how would the score change in the execution_table?

A Score would be 0

B Score would be 1

C Score would be 0.5

D Score would be negative

💡 Hint

Consider the meaning of a perfect match in evaluation logic at step 2.

Concept Snapshot

LangSmith evaluators compare model outputs to references.
They run evaluation logic to produce scores or feedback.
Use evaluator methods like evaluate_string for text.
Outputs help measure model accuracy or quality.
Scores range from 0 (bad) to 1 (perfect match).

Full Transcript

LangSmith evaluators are tools that check how well a model's output matches a reference answer. The process starts by giving the evaluator the model's prediction and the correct reference. The evaluator then runs its logic to compare these two inputs. This comparison results in a score and sometimes feedback explaining the quality. For example, a StringEvaluator compares text strings and returns a score close to 1 if they are very similar. The evaluation ends by returning this score and feedback. This helps developers understand how accurate or good their model's outputs are.