A/B testing prompt variations in LangChain - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ component_behavior

intermediate

What is the output of this LangChain prompt variation test?

Given two prompt templates tested with LangChain's A/B testing feature, what will be the output when the first prompt is selected?

LangChain

from langchain.prompts import PromptTemplate
from langchain.experimental.prompt_variations import PromptVariation

prompt_a = PromptTemplate(template="Hello, {name}!")
prompt_b = PromptTemplate(template="Hi there, {name}!")

variation = PromptVariation(variations=[prompt_a, prompt_b])

result = variation.format(name="Alice")

A "Hello, Alice!"

B "Hi there, Alice!"

C "Hello, {name}!"

D Raises TypeError because format is missing

💡 Hint

Remember that the first prompt in variations is selected by default unless specified.

The PromptVariation class picks the first prompt template by default when formatting. So it fills 'Hello, {name}!' with 'Alice'.

📝 Syntax

intermediate

Which option correctly defines two prompt variations in LangChain?

Identify the correct syntax to create two prompt variations for A/B testing using LangChain's PromptVariation.

A PromptVariation([PromptTemplate(template="Hello {name}"), PromptTemplate(template="Hi {name}")])

B PromptVariation(templates=["Hello {name}", "Hi {name}"])

C PromptVariation(variations=PromptTemplate(template="Hello {name}"), PromptTemplate(template="Hi {name}"))

D PromptVariation(variations=[PromptTemplate(template="Hello {name}"), PromptTemplate(template="Hi {name}")])

💡 Hint

Check the parameter name and how the list of prompt templates is passed.

The PromptVariation class expects a 'variations' keyword argument with a list of PromptTemplate objects.

❓ state_output

advanced

What is the value of 'selected_prompt' after running this LangChain A/B test code?

Consider this code snippet that randomly selects a prompt variation. What will be the value of 'selected_prompt' after execution?

LangChain

import random
from langchain.prompts import PromptTemplate
from langchain.experimental.prompt_variations import PromptVariation

prompt_a = PromptTemplate(template="Good morning, {name}.")
prompt_b = PromptTemplate(template="Good evening, {name}.")

variation = PromptVariation(variations=[prompt_a, prompt_b])

# Simulate random selection
selected_prompt = random.choice(variation.variations).format(name="Bob")

A Raises AttributeError because 'variations' is private

B "Good morning, Bob." or "Good evening, Bob." depending on random choice

C "Good evening, Bob." always

D "Good morning, Bob." always

💡 Hint

random.choice picks one item from the list each time it runs.

The code randomly picks one of the two prompt templates and formats it with 'Bob'. So output varies between the two greetings.

🔧 Debug

advanced

Why does this LangChain prompt variation code raise a TypeError?

Examine the code and identify why it raises a TypeError when formatting the prompt variation.

LangChain

from langchain.prompts import PromptTemplate
from langchain.experimental.prompt_variations import PromptVariation

prompt_a = PromptTemplate(template="Hello {name}")
prompt_b = PromptTemplate(template="Hi {name}")

variation = PromptVariation(variations=[prompt_a, prompt_b])

result = variation.format()

A PromptTemplate objects are not initialized correctly

B PromptVariation does not have a format method

C Missing required 'name' argument in format() call

D variations list must contain strings, not PromptTemplate objects

💡 Hint

Check the placeholders in the prompt templates and the arguments passed to format().

The prompt templates require a 'name' argument to fill the placeholder. Calling format() without arguments causes a TypeError.

🧠 Conceptual

expert

Which statement best describes the purpose of A/B testing prompt variations in LangChain?

Select the most accurate explanation of why developers use prompt variations for A/B testing in LangChain.

A To compare different prompt templates and measure which generates better responses for a task

B To automatically fix syntax errors in prompt templates during runtime

C To merge multiple prompt templates into one combined prompt for faster processing

D To cache prompt outputs to avoid repeated API calls for the same input

💡 Hint

Think about the goal of testing multiple prompts with the same input.

A/B testing prompt variations helps developers find which prompt wording works best by comparing outputs side-by-side.