A/B testing prompt variations in LangChain - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ component_behavior

intermediate

What is the output of this LangChain prompt variation test?

Given two prompt templates tested with LangChain's A/B testing feature, what will be the output when the first prompt is selected?

LangChain

from langchain.prompts import PromptTemplate
from langchain.experimental.prompt_variations import PromptVariation

prompt_a = PromptTemplate(template="Hello, {name}!")
prompt_b = PromptTemplate(template="Hi there, {name}!")

variation = PromptVariation(variations=[prompt_a, prompt_b])

result = variation.format(name="Alice")

A "Hello, Alice!"

B "Hi there, Alice!"

C "Hello, {name}!"

D Raises TypeError because format is missing

💡 Hint

Remember that the first prompt in variations is selected by default unless specified.

The PromptVariation class picks the first prompt template by default when formatting. So it fills 'Hello, {name}!' with 'Alice'.

📝 Syntax

intermediate

Which option correctly defines two prompt variations in LangChain?

Identify the correct syntax to create two prompt variations for A/B testing using LangChain's PromptVariation.

A PromptVariation([PromptTemplate(template="Hello {name}"), PromptTemplate(template="Hi {name}")])

B PromptVariation(templates=["Hello {name}", "Hi {name}"])

C PromptVariation(variations=PromptTemplate(template="Hello {name}"), PromptTemplate(template="Hi {name}"))

D PromptVariation(variations=[PromptTemplate(template="Hello {name}"), PromptTemplate(template="Hi {name}")])

💡 Hint

Check the parameter name and how the list of prompt templates is passed.

The PromptVariation class expects a 'variations' keyword argument with a list of PromptTemplate objects.

❓ state_output

advanced

What is the value of 'selected_prompt' after running this LangChain A/B test code?

Consider this code snippet that randomly selects a prompt variation. What will be the value of 'selected_prompt' after execution?

LangChain

import random
from langchain.prompts import PromptTemplate
from langchain.experimental.prompt_variations import PromptVariation

prompt_a = PromptTemplate(template="Good morning, {name}.")
prompt_b = PromptTemplate(template="Good evening, {name}.")

variation = PromptVariation(variations=[prompt_a, prompt_b])

# Simulate random selection
selected_prompt = random.choice(variation.variations).format(name="Bob")

A Raises AttributeError because 'variations' is private

B "Good morning, Bob." or "Good evening, Bob." depending on random choice

C "Good evening, Bob." always

D "Good morning, Bob." always

💡 Hint

random.choice picks one item from the list each time it runs.

The code randomly picks one of the two prompt templates and formats it with 'Bob'. So output varies between the two greetings.

🔧 Debug

advanced

Why does this LangChain prompt variation code raise a TypeError?

Examine the code and identify why it raises a TypeError when formatting the prompt variation.

LangChain

from langchain.prompts import PromptTemplate
from langchain.experimental.prompt_variations import PromptVariation

prompt_a = PromptTemplate(template="Hello {name}")
prompt_b = PromptTemplate(template="Hi {name}")

variation = PromptVariation(variations=[prompt_a, prompt_b])

result = variation.format()

A PromptTemplate objects are not initialized correctly

B PromptVariation does not have a format method

C Missing required 'name' argument in format() call

D variations list must contain strings, not PromptTemplate objects

💡 Hint

Check the placeholders in the prompt templates and the arguments passed to format().

The prompt templates require a 'name' argument to fill the placeholder. Calling format() without arguments causes a TypeError.

🧠 Conceptual

expert

Which statement best describes the purpose of A/B testing prompt variations in LangChain?

Select the most accurate explanation of why developers use prompt variations for A/B testing in LangChain.

A To compare different prompt templates and measure which generates better responses for a task

B To automatically fix syntax errors in prompt templates during runtime

C To merge multiple prompt templates into one combined prompt for faster processing

D To cache prompt outputs to avoid repeated API calls for the same input

💡 Hint

Think about the goal of testing multiple prompts with the same input.

A/B testing prompt variations helps developers find which prompt wording works best by comparing outputs side-by-side.

Practice

1. What is the main purpose of using A/B testing with prompt variations in Langchain?

easy

A. To compare different prompt versions and find the best one

B. To speed up the execution of a single prompt

C. To combine multiple prompts into one

D. To automatically fix errors in prompts

Solution

1. Step 1: Understand A/B testing concept

   A/B testing means comparing two or more versions to see which works better.

2. Step 2: Apply to prompt variations

   In Langchain, this means running different prompt templates and comparing their outputs.

3. Final Answer:

   To compare different prompt versions and find the best one -> Option A

4. Quick Check:

   A/B testing = Compare versions [OK]

Hint: A/B testing means comparing versions to pick the best [OK]

Common Mistakes:

• Thinking A/B testing speeds up prompts
• Believing it merges prompts automatically
• Assuming it fixes prompt errors

2. Which of the following is the correct way to create two prompt variations for A/B testing in Langchain using the 'template=' keyword argument for both PromptTemplates?

easy

A. prompt1 = PromptTemplate('Hello {name}'); prompt2 = PromptTemplate(template='Hi {name}')

B. prompt1 = PromptTemplate('Hello {name}'); prompt2 = PromptTemplate('Hi {name}')

C. prompt1 = PromptTemplate(template='Hello {name}'); prompt2 = PromptTemplate(template='Hi {name}')

D. prompt1 = PromptTemplate(template='Hello {name}'); prompt2 = PromptTemplate('Hi {name}')

Solution

1. Step 1: Check PromptTemplate syntax

   PromptTemplate uses the named argument 'template' to define the prompt string.

2. Step 2: Verify both prompts use correct syntax

   Only prompt1 = PromptTemplate(template='Hello {name}'); prompt2 = PromptTemplate(template='Hi {name}') uses PromptTemplate(template='...') for both prompts correctly.

3. Final Answer:

   prompt1 = PromptTemplate(template='Hello {name}'); prompt2 = PromptTemplate(template='Hi {name}') -> Option C

4. Quick Check:

   Use template= keyword for PromptTemplate [OK]

Hint: PromptTemplate needs template='...' argument [OK]

Common Mistakes:

• Omitting the 'template=' keyword
• Mixing positional and keyword arguments
• Using incorrect string syntax

3. Given the code below, what will be the output of print(results)?

from langchain import PromptTemplate
prompt1 = PromptTemplate(template='Hello {name}')
prompt2 = PromptTemplate(template='Hi {name}')
inputs = {'name': 'Alice'}
results = [prompt1.format(**inputs), prompt2.format(**inputs)]
print(results)

medium

A. ['Hello Alice', 'Hi Alice']

B. ['Hello {name}', 'Hi {name}']

C. ['Hello', 'Hi']

D. Error: format method not found

Solution

1. Step 1: Understand PromptTemplate.format()

   The format method replaces placeholders like {name} with values from inputs.

2. Step 2: Apply inputs to both prompts

   Both prompts get 'Alice' for {name}, so outputs are 'Hello Alice' and 'Hi Alice'.

3. Final Answer:

   ['Hello Alice', 'Hi Alice'] -> Option A

4. Quick Check:

   format() replaces placeholders correctly [OK]

Hint: format() fills placeholders with input values [OK]

Common Mistakes:

• Thinking format() returns template string unchanged
• Expecting placeholders to remain in output
• Assuming format() method does not exist

4. Identify the error in this A/B testing code snippet:

from langchain import PromptTemplate
prompt1 = PromptTemplate(template='Hello {name}')
prompt2 = PromptTemplate(template='Hi {name}')
inputs = {'name': 'Bob'}
results = [prompt1.format(inputs), prompt2.format(inputs)]
print(results)

medium

A. PromptTemplate missing template argument

B. Using format() without unpacking inputs dictionary

C. inputs dictionary missing required key

D. print statement syntax error

Solution

1. Step 1: Check how format() is called

   format() expects keyword arguments, so inputs must be unpacked with **inputs.

2. Step 2: Identify the error

   Code passes inputs as a single dict argument, causing a TypeError.

3. Final Answer:

   Using format() without unpacking inputs dictionary -> Option B

4. Quick Check:

   Use **inputs to unpack dict for format() [OK]

Hint: Always unpack dict with ** when calling format() [OK]

Common Mistakes:

• Passing dict directly instead of unpacking
• Forgetting to import PromptTemplate
• Using wrong print syntax

5. You want to run A/B testing on three prompt variations and select the best output based on a scoring function. Which approach correctly implements this in Langchain?

hard

A. Use a loop to create prompts but do not run format(), just score the templates

B. Create one PromptTemplate with all variations combined, run format() once, then score the single output

C. Run format() on one prompt, then copy the output three times and score them

D. Create three PromptTemplate objects, run format() on each with inputs, then apply the scoring function to outputs and pick the highest score

Solution

1. Step 1: Understand A/B testing with multiple prompts

   You need separate prompt templates for each variation to test them individually.

2. Step 2: Run each prompt with the same inputs and score outputs

   Format each prompt with inputs, then apply scoring to compare results.

3. Step 3: Select the best output based on scores

   Pick the output with the highest score as the best prompt result.

4. Final Answer:

   Create three PromptTemplate objects, run format() on each with inputs, then apply the scoring function to outputs and pick the highest score -> Option D

5. Quick Check:

   Separate prompts + score outputs = best choice [OK]

Hint: Run all prompts, score outputs, pick highest score [OK]

Common Mistakes:

• Combining prompts into one string
• Scoring templates instead of outputs
• Not running format() before scoring