Custom evaluation metrics in LangChain - Performance & Optimization

Performance: Custom evaluation metrics

MEDIUM IMPACT

Custom evaluation metrics affect the speed and responsiveness of model output evaluation during runtime.

Evaluating model outputs with custom metrics during runtime

LangChain

def fast_metric(output, reference):
    output_set = set(output)
    reference_set = set(reference)
    score = len(output_set & reference_set) / len(reference_set)
    return score

Uses set operations which are faster and avoid nested loops, reducing evaluation time.

📈 Performance GainReduces blocking to under 10 ms, improving interaction responsiveness

Evaluating model outputs with custom metrics during runtime

LangChain

def slow_metric(output, reference):
    # Complex nested loops and heavy computations
    score = 0
    for o in output:
        for r in reference:
            if o == r:
                score += 1
    return score / len(reference)

This metric uses nested loops causing slow evaluation especially with large outputs, blocking UI updates.

📉 Performance CostBlocks rendering for 100+ ms on large inputs, causing input delay

Performance Comparison

Pattern	DOM Operations	Reflows	Paint Cost	Verdict
Nested loops metric	Minimal	0	High due to blocking JS	[X] Bad
Set operations metric	Minimal	0	Low due to fast JS	[OK] Good

Rendering Pipeline

Custom evaluation metrics run after model output generation and before UI update, affecting the interaction to next paint phase.

→JavaScript Execution

→Layout

→Paint

⚠️ BottleneckJavaScript Execution due to heavy synchronous computations

Core Web Vital Affected

INP

Custom evaluation metrics affect the speed and responsiveness of model output evaluation during runtime.

Optimization Tips

1Avoid nested loops in metric calculations to reduce blocking.

2Use efficient data structures like sets for faster evaluation.

3Run heavy computations asynchronously to keep UI responsive.

Performance Quiz - 3 Questions

What is the main performance issue with using nested loops in custom evaluation metrics?

A They increase DOM node count

B They trigger multiple reflows

C They cause blocking JavaScript execution slowing interaction

D They increase CSS selector complexity

DevTools: Performance

How to check: Record a performance profile while running the evaluation metric; look for long scripting tasks.

What to look for: Long blocking JavaScript tasks indicate slow metric evaluation affecting responsiveness.