Context formatting and injection in LangChain - Performance & Optimization

Performance: Context formatting and injection

MEDIUM IMPACT

This concept affects how quickly the language model can generate responses by controlling prompt size and complexity, impacting initial load and interaction speed.

Injecting context into prompts for language model queries

LangChain

formatted_context = format_context(large_documents, max_tokens=500)
prompt = f"Answer based on context: {formatted_context}"
response = llm.generate(prompt)

Formats and truncates context to essential parts, reducing prompt size and speeding up model generation.

📈 Performance GainReduces prompt tokens by 70%, improving response time and lowering compute usage

Injecting context into prompts for language model queries

LangChain

context = "".join(large_documents)
prompt = f"Answer based on context: {context}"
response = llm.generate(prompt)

Injecting large unformatted context causes long prompt strings, increasing token count and slowing model response.

📉 Performance CostIncreases prompt size by hundreds of tokens, causing slower response and higher compute cost

Performance Comparison

Pattern	Prompt Size	Token Count	Response Latency	Verdict
Inject full raw context	Large (many KB)	High (1000+ tokens)	Slow (seconds delay)	[X] Bad
Inject formatted, truncated context	Small (few KB)	Low (few hundred tokens)	Fast (sub-second delay)	[OK] Good

Rendering Pipeline

Context formatting and injection affects the prompt construction stage before sending data to the language model API. Larger prompts increase token processing time and network payload size.

→Prompt Construction

→Network Transfer

→Model Inference

⚠️ BottleneckModel Inference due to larger token input

Core Web Vital Affected

INP

This concept affects how quickly the language model can generate responses by controlling prompt size and complexity, impacting initial load and interaction speed.

Optimization Tips

1Always limit context size to essential information before injection.

2Format context to remove unnecessary data and reduce tokens.

3Avoid injecting raw large documents directly into prompts.

Performance Quiz - 3 Questions

What is the main performance impact of injecting large unformatted context into a language model prompt?

A Improves model accuracy without speed impact

B Reduces network payload size

C Increases token count, slowing model response

D Speeds up prompt construction

DevTools: Network

How to check: Open DevTools, go to Network tab, filter requests to the language model API, inspect the request payload size and timing.

What to look for: Look for large request payloads indicating big prompts and long response times showing slow model inference.

Practice

1. What is the main purpose of context formatting in Langchain?

easy

A. To create database connections

B. To execute code asynchronously

C. To prepare prompts with placeholders for dynamic data

D. To store user credentials securely

Solution

1. Step 1: Understand context formatting

   Context formatting is about creating prompt templates with placeholders that can be filled later.

2. Step 2: Identify the purpose

   This allows dynamic injection of data into prompts, making AI responses relevant.

3. Final Answer:

   To prepare prompts with placeholders for dynamic data -> Option C

4. Quick Check:

   Context formatting = placeholders in prompts [OK]

Hint: Think of placeholders as empty boxes to fill later [OK]

Common Mistakes:

• Confusing context formatting with code execution
• Thinking it stores credentials
• Mixing it with database operations

2. Which of the following is the correct way to inject a variable user_name into a Langchain prompt template?

easy

A. prompt_template.inject(user_name)

B. prompt_template.append(user_name)

C. prompt_template.fill(user_name)

D. prompt_template.format(user_name=user_name)

Solution

1. Step 1: Recall Langchain prompt syntax

   Langchain uses the format() method to fill placeholders in prompt templates.

2. Step 2: Match method to injection

   Only format(user_name=user_name) correctly injects the variable into the placeholder.

3. Final Answer:

   prompt_template.format(user_name=user_name) -> Option D

4. Quick Check:

   Use format() to inject variables [OK]

Hint: Use .format() to fill placeholders in templates [OK]

Common Mistakes:

• Using non-existent methods like inject() or fill()
• Trying to append variables instead of formatting
• Forgetting to name the variable in format()

3. Given this code snippet:

template = "Hello, {name}! Today is {day}."
prompt = template.format(name="Alice", day="Monday")
print(prompt)

What will be the output?

medium

A. Hello, Alice! Today is Monday.

B. Hello, name! Today is day.

C. Hello, {name}! Today is {day}.

D. Error: Missing placeholders

Solution

1. Step 1: Understand the template string

   The template has placeholders {name} and {day} to be replaced by actual values.

2. Step 2: Apply the format method

   Calling format(name="Alice", day="Monday") replaces placeholders with given values.

3. Final Answer:

   Hello, Alice! Today is Monday. -> Option A

4. Quick Check:

   Placeholders replaced by values = output string [OK]

Hint: format() replaces placeholders with given values [OK]

Common Mistakes:

• Printing the template without formatting
• Confusing placeholder names with literal text
• Expecting an error when placeholders are correctly filled

4. Identify the error in this Langchain prompt injection code:

template = "Welcome, {user}!"
prompt = template.format(user)
print(prompt)

medium

A. Incorrect placeholder syntax in template

B. Missing variable name in format() call

C. Using print() instead of return

D. Template string should use double braces {{}}

Solution

1. Step 1: Check format() usage

   The format method requires named arguments matching placeholders or positional arguments in order.

2. Step 2: Identify missing variable name

   Here, format(user) passes a positional argument but the placeholder expects a named argument {user}.

3. Final Answer:

   Missing variable name in format() call -> Option B

4. Quick Check:

   Named placeholders need named arguments in format() [OK]

Hint: Match placeholder names with named format() arguments [OK]

Common Mistakes:

• Passing positional argument without naming it
• Thinking placeholder syntax is wrong
• Confusing print() with return statements

5. You want to create a prompt that greets a user and mentions their favorite color using Langchain. Which code correctly formats and injects the context for user_name = "Bob" and color = "blue"?

hard

A. template = "Hi {user_name}, your favorite color is {color}."
prompt = template.format(user_name=user_name, color=color)

B. template = "Hi {user}, your favorite color is {color}."
prompt = template.format(user_name=user_name, color=color)

C. template = "Hi {user_name}, your favorite color is {color}."
prompt = template.format(user=user_name, color=color)

D. template = "Hi {user_name}, your favorite color is {color}."
prompt = template.fill(user_name, color)

Solution

1. Step 1: Match placeholders with variable names

   The template uses placeholders {user_name} and {color}, so format() must use these exact names.

2. Step 2: Check format() arguments

   template = "Hi {user_name}, your favorite color is {color}." prompt = template.format(user_name=user_name, color=color) correctly passes named arguments matching placeholders: user_name=user_name, color=color.

3. Step 3: Eliminate incorrect options

   template = "Hi {user}, your favorite color is {color}." prompt = template.format(user_name=user_name, color=color) uses {user} placeholder mismatching the user_name= argument. template = "Hi {user_name}, your favorite color is {color}." prompt = template.format(user=user_name, color=color) mismatches placeholder and argument names. template = "Hi {user_name}, your favorite color is {color}." prompt = template.fill(user_name, color) uses non-existent fill() method.

4. Final Answer:

   template = "Hi {user_name}, your favorite color is {color}." prompt = template.format(user_name=user_name, color=color) -> Option A

5. Quick Check:

   Placeholder names must match format() argument names [OK]

Hint: Keep placeholder and format() argument names identical [OK]

Common Mistakes:

• Mismatching placeholder and argument names
• Using wrong method like fill()
• Ignoring placeholder names in template