A/B testing prompt variations in LangChain - Mini Project: Build & Apply

A/B Testing Prompt Variations with LangChain

📖 Scenario: You are building a chatbot that tries different prompt styles to see which one gets better answers from a language model.This helps you find the best way to ask questions.

🎯 Goal: Create a LangChain setup that holds two prompt variations, sends them to the language model, and compares the responses.

📋 What You'll Learn

Create a dictionary called prompt_variations with two exact prompt strings.

Create a variable called model_name set to 'gpt-4o-mini'.

Use a for loop with variables variation_name and prompt_text to iterate over prompt_variations.items().

Create a LangChain PromptTemplate and LLMChain for each prompt variation and store the responses in a dictionary called responses.

💡 Why This Matters

🌍 Real World

A/B testing prompt variations helps chatbot developers find the best way to ask questions to get clearer or more useful answers from AI models.

💼 Career

This skill is useful for AI developers, prompt engineers, and anyone working with language models to optimize user interactions.

1

Create the prompt variations dictionary

Create a dictionary called prompt_variations with these exact entries: 'variation_a': 'Explain the benefits of exercise in simple terms.' and 'variation_b': 'List three reasons why exercise is good for health.'

LangChain

# Create the prompt_variations dictionary with two exact prompts
# Your code here

Hint

Use curly braces to create a dictionary with two keys: 'variation_a' and 'variation_b'.

2

Set the model name variable

Create a variable called model_name and set it to the string 'gpt-4o-mini'.

LangChain

prompt_variations = {
    'variation_a': 'Explain the benefits of exercise in simple terms.',
    'variation_b': 'List three reasons why exercise is good for health.'
}
# Create the model_name variable and set it to 'gpt-4o-mini'
# Your code here

Hint

Just assign the string 'gpt-4o-mini' to the variable model_name.

3

Create the LangChain prompt templates and chains

Use a for loop with variables variation_name and prompt_text to iterate over prompt_variations.items(). Inside the loop, create a PromptTemplate with input_variables=[] and template=prompt_text. Then create an LLMChain with llm=ChatOpenAI(model_name=model_name) and the prompt template. Store the chain in a dictionary called chains with keys as variation_name.

LangChain

prompt_variations = {
    'variation_a': 'Explain the benefits of exercise in simple terms.',
    'variation_b': 'List three reasons why exercise is good for health.'
}

model_name = 'gpt-4o-mini'

from langchain.chat_models import ChatOpenAI
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate

# Create a dictionary called chains
# Use a for loop to create PromptTemplate and LLMChain for each prompt
# Your code here

Hint

Use a for loop to create prompt templates and chains, then save them in the chains dictionary.

4

Run the chains and collect responses

Create an empty dictionary called responses. Use a for loop with variables variation_name and chain to iterate over chains.items(). Inside the loop, call chain.run() and store the result in responses[variation_name].

LangChain

prompt_variations = {
    'variation_a': 'Explain the benefits of exercise in simple terms.',
    'variation_b': 'List three reasons why exercise is good for health.'
}

model_name = 'gpt-4o-mini'

from langchain.chat_models import ChatOpenAI
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate

chains = {}
for variation_name, prompt_text in prompt_variations.items():
    prompt = PromptTemplate(input_variables=[], template=prompt_text)
    chain = LLMChain(llm=ChatOpenAI(model_name=model_name), prompt=prompt)
    chains[variation_name] = chain

# Create a dictionary called responses
# Use a for loop to run each chain and save the output
# Your code here

Hint

Run each chain and save the answers in the responses dictionary using a for loop.

Practice

1. What is the main purpose of using A/B testing with prompt variations in Langchain?

easy

A. To compare different prompt versions and find the best one

B. To speed up the execution of a single prompt

C. To combine multiple prompts into one

D. To automatically fix errors in prompts

Solution

1. Step 1: Understand A/B testing concept

   A/B testing means comparing two or more versions to see which works better.

2. Step 2: Apply to prompt variations

   In Langchain, this means running different prompt templates and comparing their outputs.

3. Final Answer:

   To compare different prompt versions and find the best one -> Option A

4. Quick Check:

   A/B testing = Compare versions [OK]

Hint: A/B testing means comparing versions to pick the best [OK]

Common Mistakes:

• Thinking A/B testing speeds up prompts
• Believing it merges prompts automatically
• Assuming it fixes prompt errors

2. Which of the following is the correct way to create two prompt variations for A/B testing in Langchain using the 'template=' keyword argument for both PromptTemplates?

easy

A. prompt1 = PromptTemplate('Hello {name}'); prompt2 = PromptTemplate(template='Hi {name}')

B. prompt1 = PromptTemplate('Hello {name}'); prompt2 = PromptTemplate('Hi {name}')

C. prompt1 = PromptTemplate(template='Hello {name}'); prompt2 = PromptTemplate(template='Hi {name}')

D. prompt1 = PromptTemplate(template='Hello {name}'); prompt2 = PromptTemplate('Hi {name}')

Solution

1. Step 1: Check PromptTemplate syntax

   PromptTemplate uses the named argument 'template' to define the prompt string.

2. Step 2: Verify both prompts use correct syntax

   Only prompt1 = PromptTemplate(template='Hello {name}'); prompt2 = PromptTemplate(template='Hi {name}') uses PromptTemplate(template='...') for both prompts correctly.

3. Final Answer:

   prompt1 = PromptTemplate(template='Hello {name}'); prompt2 = PromptTemplate(template='Hi {name}') -> Option C

4. Quick Check:

   Use template= keyword for PromptTemplate [OK]

Hint: PromptTemplate needs template='...' argument [OK]

Common Mistakes:

• Omitting the 'template=' keyword
• Mixing positional and keyword arguments
• Using incorrect string syntax

3. Given the code below, what will be the output of print(results)?

from langchain import PromptTemplate
prompt1 = PromptTemplate(template='Hello {name}')
prompt2 = PromptTemplate(template='Hi {name}')
inputs = {'name': 'Alice'}
results = [prompt1.format(**inputs), prompt2.format(**inputs)]
print(results)

medium

A. ['Hello Alice', 'Hi Alice']

B. ['Hello {name}', 'Hi {name}']

C. ['Hello', 'Hi']

D. Error: format method not found

Solution

1. Step 1: Understand PromptTemplate.format()

   The format method replaces placeholders like {name} with values from inputs.

2. Step 2: Apply inputs to both prompts

   Both prompts get 'Alice' for {name}, so outputs are 'Hello Alice' and 'Hi Alice'.

3. Final Answer:

   ['Hello Alice', 'Hi Alice'] -> Option A

4. Quick Check:

   format() replaces placeholders correctly [OK]

Hint: format() fills placeholders with input values [OK]

Common Mistakes:

• Thinking format() returns template string unchanged
• Expecting placeholders to remain in output
• Assuming format() method does not exist

4. Identify the error in this A/B testing code snippet:

from langchain import PromptTemplate
prompt1 = PromptTemplate(template='Hello {name}')
prompt2 = PromptTemplate(template='Hi {name}')
inputs = {'name': 'Bob'}
results = [prompt1.format(inputs), prompt2.format(inputs)]
print(results)

medium

A. PromptTemplate missing template argument

B. Using format() without unpacking inputs dictionary

C. inputs dictionary missing required key

D. print statement syntax error

Solution

1. Step 1: Check how format() is called

   format() expects keyword arguments, so inputs must be unpacked with **inputs.

2. Step 2: Identify the error

   Code passes inputs as a single dict argument, causing a TypeError.

3. Final Answer:

   Using format() without unpacking inputs dictionary -> Option B

4. Quick Check:

   Use **inputs to unpack dict for format() [OK]

Hint: Always unpack dict with ** when calling format() [OK]

Common Mistakes:

• Passing dict directly instead of unpacking
• Forgetting to import PromptTemplate
• Using wrong print syntax

5. You want to run A/B testing on three prompt variations and select the best output based on a scoring function. Which approach correctly implements this in Langchain?

hard

A. Use a loop to create prompts but do not run format(), just score the templates

B. Create one PromptTemplate with all variations combined, run format() once, then score the single output

C. Run format() on one prompt, then copy the output three times and score them

D. Create three PromptTemplate objects, run format() on each with inputs, then apply the scoring function to outputs and pick the highest score

Solution

1. Step 1: Understand A/B testing with multiple prompts

   You need separate prompt templates for each variation to test them individually.

2. Step 2: Run each prompt with the same inputs and score outputs

   Format each prompt with inputs, then apply scoring to compare results.

3. Step 3: Select the best output based on scores

   Pick the output with the highest score as the best prompt result.

4. Final Answer:

   Create three PromptTemplate objects, run format() on each with inputs, then apply the scoring function to outputs and pick the highest score -> Option D

5. Quick Check:

   Separate prompts + score outputs = best choice [OK]

Hint: Run all prompts, score outputs, pick highest score [OK]

Common Mistakes:

• Combining prompts into one string
• Scoring templates instead of outputs
• Not running format() before scoring