Comparing prompt versions in LangChain - Step-by-Step Execution

Concept Flow - Comparing prompt versions

Define Prompt Version 1

↓

Define Prompt Version 2

↓

Input User Query

↓

Run Prompt Version 1 with Query

↓

Run Prompt Version 2 with Query

↓

Compare Outputs

↓

Decide Best Prompt Version

This flow shows how two prompt versions are defined, run with the same input, and their outputs compared to choose the best.

Execution Sample

LangChain

from langchain import PromptTemplate

prompt_v1 = PromptTemplate(template="Hello {name}!")
prompt_v2 = PromptTemplate(template="Hi there, {name}!")

input_data = {"name": "Alice"}

output_v1 = prompt_v1.format(**input_data)
output_v2 = prompt_v2.format(**input_data)

print(output_v1)
print(output_v2)

This code creates two prompt versions and formats them with the same input to compare their outputs.

Execution Table

Step	Action	Input	Output	Notes
1	Define prompt_v1	template="Hello {name}!"	PromptTemplate object created	Stores first prompt version
2	Define prompt_v2	template="Hi there, {name}!"	PromptTemplate object created	Stores second prompt version
3	Set input_data	{"name": "Alice"}	Dictionary with name key	Input for formatting
4	Format prompt_v1	input_data	"Hello Alice!"	First prompt output
5	Format prompt_v2	input_data	"Hi there, Alice!"	Second prompt output
6	Print output_v1	"Hello Alice!"	Displays: Hello Alice!	Output from prompt_v1
7	Print output_v2	"Hi there, Alice!"	Displays: Hi there, Alice!	Output from prompt_v2
8	Compare outputs	"Hello Alice!" vs "Hi there, Alice!"	Decide which sounds better	Comparison step
9	End	-	-	Comparison complete

💡 All steps executed to compare prompt outputs for same input.

Variable Tracker

Variable	Start	After Step 3	After Step 4	After Step 5	Final
prompt_v1	undefined	PromptTemplate(template="Hello {name}!")	Same	Same	Same
prompt_v2	undefined	PromptTemplate(template="Hi there, {name}!")	Same	Same	Same
input_data	undefined	{"name": "Alice"}	Same	Same	Same
output_v1	undefined	undefined	"Hello Alice!"	"Hello Alice!"	"Hello Alice!"
output_v2	undefined	undefined	undefined	"Hi there, Alice!"	"Hi there, Alice!"

Key Moments - 3 Insights

Why do both prompt versions use the same input_data?

Using the same input_data ensures a fair comparison of outputs, as shown in execution_table rows 3, 4, and 5.

What happens if the templates have different placeholders?

If placeholders differ, formatting will fail or produce incorrect output. Here, both use {name}, so formatting succeeds (rows 4 and 5).

How do we decide which prompt version is better?

We compare the outputs (row 8) to see which phrasing fits our needs or sounds better.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output of prompt_v1 at step 4?

A "Hi there, Alice!"

B "Hello Alice!"

C "Hello {name}!"

D An error

💡 Hint

Check the Output column at step 4 in the execution_table.

At which step do we compare the outputs of both prompt versions?

A Step 5

B Step 3

C Step 8

D Step 7

💡 Hint

Look for the row mentioning 'Compare outputs' in the execution_table.

If input_data changed to {"name": "Bob"}, what would output_v2 be at step 5?

A "Hi there, Bob!"

B "Hi there, Alice!"

C "Hello Bob!"

D "Hello Alice!"

💡 Hint

Refer to variable_tracker and how formatting replaces {name} with input_data value.

Concept Snapshot

Comparing prompt versions:
- Define multiple PromptTemplate objects with different templates.
- Use the same input data to format each prompt.
- Run each prompt with input to get outputs.
- Compare outputs to choose the best prompt.
- Ensures fair testing of prompt wording.

Full Transcript

This visual execution shows how to compare two prompt versions in langchain. First, we create two PromptTemplate objects with different text templates. Then, we prepare the same input data dictionary with a name key. Next, we format each prompt with this input to produce output strings. We print both outputs to see their differences. Finally, we compare these outputs to decide which prompt version sounds better or fits our needs. Tracking variables shows prompt templates stay the same, input data is constant, and outputs change after formatting. Key moments clarify why same input is needed and how to handle placeholders. The quiz tests understanding of outputs and comparison steps.

Practice

1. What is the main purpose of comparing different prompt versions in Langchain?

easy

A. To find the best wording that improves AI task results

B. To increase the number of API calls

C. To reduce the size of the prompt template

D. To change the programming language used

Solution

1. Step 1: Understand the goal of prompt comparison

   Comparing prompt versions helps identify which wording or structure yields better AI responses.

2. Step 2: Eliminate unrelated options

   Increasing API calls, reducing prompt size, or changing language do not relate to improving prompt effectiveness.

3. Final Answer:

   To find the best wording that improves AI task results -> Option A

4. Quick Check:

   Comparing prompts = find best wording [OK]

Hint: Focus on improving AI output quality, not technical details [OK]

Common Mistakes:

• Thinking prompt comparison reduces API calls
• Confusing prompt size with prompt quality
• Assuming language change is the goal

2. Which of the following is the correct way to create a PromptTemplate in Langchain?

easy

A. PromptTemplate(prompt="Hello {name}", args=["name"])

B. PromptTemplate(name="Hello {name}", variables=["name"])

C. PromptTemplate(text="Hello {name}", inputs=["name"])

D. PromptTemplate(template="Hello {name}", input_variables=["name"])

Solution

1. Step 1: Recall PromptTemplate syntax

   The correct constructor uses 'template' for the prompt text and 'input_variables' for placeholders.

2. Step 2: Check each option

   Only PromptTemplate(template="Hello {name}", input_variables=["name"]) uses 'template' and 'input_variables' correctly; others use wrong parameter names.

3. Final Answer:

   PromptTemplate(template="Hello {name}", input_variables=["name"]) -> Option D

4. Quick Check:

   Correct parameters = template + input_variables [OK]

Hint: Remember: 'template' and 'input_variables' are required keys [OK]

Common Mistakes:

• Using 'name' instead of 'template' for prompt text
• Using 'variables' instead of 'input_variables'
• Confusing parameter names

3. Given the code below, what will be printed?

from langchain import PromptTemplate

prompt_v1 = PromptTemplate(template="Hello, {name}!", input_variables=["name"])
prompt_v2 = PromptTemplate(template="Hi {name}, how are you?", input_variables=["name"])

print(prompt_v1.format(name="Alice"))
print(prompt_v2.format(name="Alice"))

medium

A. Hello Alice!
Hi Alice, how are you?

B. Hello, Alice!
Hi Alice, how are you?

C. Hello, {name}!
Hi {name}, how are you?

D. Error: Missing input variable

Solution

1. Step 1: Understand PromptTemplate.format()

   The format method replaces placeholders with provided values, here 'name' is 'Alice'.

2. Step 2: Apply formatting to each prompt

   prompt_v1 becomes "Hello, Alice!" and prompt_v2 becomes "Hi Alice, how are you?".

3. Final Answer:

   Hello, Alice! Hi Alice, how are you? -> Option B

4. Quick Check:

   Formatted prompts show replaced names [OK]

Hint: Format replaces {name} with 'Alice' exactly [OK]

Common Mistakes:

• Ignoring commas or punctuation in output
• Printing raw template without formatting
• Assuming error without missing inputs

4. What is the error in the following code snippet?

from langchain import PromptTemplate

prompt = PromptTemplate(template="Hello, {user}!")
print(prompt.format(name="Bob"))

medium

A. Missing input_variables parameter in PromptTemplate

B. PromptTemplate cannot be imported from langchain

C. Using 'name' instead of 'user' in format call

D. No error, code runs fine

Solution

1. Step 1: Check PromptTemplate parameters

   While 'input_variables' is recommended, it is optional if placeholders are in template.

2. Step 2: Check format call variables

   The template expects 'user' but format is called with 'name', causing a KeyError.

3. Final Answer:

   Using 'name' instead of 'user' in format call -> Option C

4. Quick Check:

   Format keys must match template placeholders [OK]

Hint: Match format keys exactly to template placeholders [OK]

Common Mistakes:

• Assuming missing input_variables causes error
• Thinking import is wrong
• Ignoring variable name mismatch

5. You want to compare two prompt versions to see which generates a more polite greeting. You have these prompts:

prompt_v1 = PromptTemplate(template="Hey {name}, what's up?", input_variables=["name"])
prompt_v2 = PromptTemplate(template="Good day, {name}. How do you do?", input_variables=["name"])

Which approach best helps you compare their effectiveness?

hard

A. Format both prompts with the same name and print outputs side-by-side for review

B. Use only prompt_v1 since it is shorter and simpler

C. Change the input variable names to different ones for each prompt

D. Run prompt_v2 without formatting to see the raw template

Solution

1. Step 1: Understand comparison goal

   You want to see which prompt wording sounds more polite for the same input.

2. Step 2: Use consistent input and print both outputs

   Formatting both prompts with the same name and printing outputs side-by-side lets you compare wording clearly.

3. Final Answer:

   Format both prompts with the same name and print outputs side-by-side for review -> Option A

4. Quick Check:

   Compare outputs side-by-side for best prompt [OK]

Hint: Print both formatted prompts together to compare easily [OK]

Common Mistakes:

• Choosing only one prompt without comparison
• Changing input variable names inconsistently
• Not formatting prompts before comparing