Regression testing for chains in LangChain

Introduction

Regression testing helps make sure that changes in your chain do not break its expected behavior. It checks that the chain still works as before.

After updating or improving a chain to confirm it still gives correct results.

When adding new features to a chain to ensure old features still work.

Before releasing a chain to users to avoid unexpected errors.

When fixing bugs in a chain to verify the fix and no new problems appear.

Syntax

LangChain

from langchain.chains import SomeChain
from langchain.chains.regression import RegressionTest

# Create your chain
chain = SomeChain(...)

# Define test inputs and expected outputs
test_cases = [
    {"input": {...}, "expected_output": {...}},
    ...
]

# Create regression test object
regression_test = RegressionTest(chain=chain, test_cases=test_cases)

# Run regression tests
results = regression_test.run()

# Check results
print(results)

You need to provide clear input and expected output pairs for testing.

The RegressionTest class runs the chain on inputs and compares outputs automatically.

Examples

A simple test case with input text and expected chain response.

LangChain

test_cases = [
    {"input": {"text": "Hello"}, "expected_output": {"response": "Hi there!"}}
]

Run regression tests and print whether each test passed or failed.

LangChain

regression_test = RegressionTest(chain=my_chain, test_cases=test_cases)
results = regression_test.run()
print(results)

Sample Program

This program creates a simple chain that uses an LLM to say hello to a name. It then runs regression tests to check if the chain output matches expected text.

LangChain

from langchain.chains import LLMChain
from langchain.llms import OpenAI
from langchain.chains.regression import RegressionTest

# Create a simple chain that echoes input text
llm = OpenAI(temperature=0)
prompt = "Say hello to {name}."
chain = LLMChain(llm=llm, prompt=prompt)

# Define test cases
test_cases = [
    {"input": {"name": "Alice"}, "expected_output": {"text": "Say hello to Alice."}},
    {"input": {"name": "Bob"}, "expected_output": {"text": "Say hello to Bob."}}
]

# Create regression test
regression_test = RegressionTest(chain=chain, test_cases=test_cases)

# Run tests
results = regression_test.run()

# Print results
print(results)

Important Notes

Regression tests help catch bugs early by comparing current outputs to saved expected outputs.

Keep test cases small and focused for easier debugging.

Update expected outputs only when you intentionally change chain behavior.

Summary

Regression testing checks that chains keep working after changes.

It uses input-output pairs to verify chain results.

Running regression tests helps maintain chain reliability.