Combining retrieved context with LLM in Prompt Engineering / GenAI - Full Explanation

Introduction

Imagine trying to answer a question without all the facts you need. This problem happens when language models try to generate answers but lack specific information. Combining retrieved context with a language model helps solve this by giving the model extra facts to work with.

Explanation

Retrieval of Relevant Information

Before the language model answers, it searches a large collection of documents or data to find the most relevant pieces of information. This step ensures the model has facts related to the question or task. The retrieval can be done using keywords, similarity searches, or other methods.

Finding the right information first is essential to give the model useful context.

Feeding Context to the Language Model

The retrieved information is then added to the input given to the language model. This extra context helps the model understand the question better and produce more accurate and detailed answers. The model uses both the question and the added facts to generate its response.

Adding relevant facts to the model's input improves answer quality.

Balancing Context Length and Model Limits

Language models have limits on how much text they can process at once. It is important to select and shorten the retrieved context so it fits within these limits without losing key information. This balance ensures the model can use the context effectively without being overwhelmed.

Careful selection of context keeps the input manageable and useful.

Improving Accuracy and Trustworthiness

By combining retrieved facts with the model's language skills, the answers become more accurate and trustworthy. The model is less likely to guess or make up information because it can rely on real data. This approach is especially helpful for complex or specialized questions.

Using real facts with the model reduces errors and increases trust.

Real World Analogy

Imagine you are asked a tricky question during a quiz. Instead of guessing, you quickly look up the answer in a trusted book and then explain it. This way, your answer is both confident and correct because you combined your speaking skills with the right information.

Retrieval of Relevant Information → Looking up the answer in a trusted book before speaking

Feeding Context to the Language Model → Using the book's information to help explain the answer clearly

Balancing Context Length and Model Limits → Choosing only the important parts of the book to read quickly

Improving Accuracy and Trustworthiness → Giving a confident answer based on real facts, not guessing

Diagram

Diagram

┌───────────────────────────┐
│      User Question        │
└────────────┬──────────────┘
             │
             ▼
┌───────────────────────────┐
│  Retrieval System Searches │
│  Documents for Context     │
└────────────┬──────────────┘
             │
             ▼
┌───────────────────────────┐
│  Retrieved Relevant Text  │
└────────────┬──────────────┘
             │
             ▼
┌───────────────────────────┐
│  Combine Question + Context│
│  Input to Language Model   │
└────────────┬──────────────┘
             │
             ▼
┌───────────────────────────┐
│  Language Model Generates  │
│  Answer Using Context      │
└───────────────────────────┘

This diagram shows how a user question leads to retrieving context, which is combined with the question and fed into the language model to generate an answer.

Key Facts

Context Retrieval → The process of finding relevant information from a data source to help answer a question.

Language Model Input → The combined text of the user question and retrieved context given to the model.

Token Limit → The maximum amount of text a language model can process at one time.

Answer Accuracy → How correct and reliable the model's response is.

Common Confusions

Believing the language model already knows all facts and does not need extra context.

Believing the language model already knows all facts and does not need extra context. Language models generate text based on patterns learned from data but do not have up-to-date or complete knowledge; retrieved context fills this gap.

Thinking more context always improves answers regardless of length.

Thinking more context always improves answers regardless of length. Too much context can exceed model limits or include irrelevant details, which can confuse the model and reduce answer quality.

Summary

Combining retrieved context with a language model helps answer questions more accurately by providing relevant facts.

The process involves finding useful information, adding it to the model's input, and managing input size carefully.

This approach reduces guessing and improves trust in the model's responses.

Practice

1. Why do we combine retrieved context with a large language model (LLM)?

easy

A. To give the model extra information it did not learn before

B. To make the model run faster

C. To reduce the size of the model

D. To replace the model's training data

Solution

1. Step 1: Understand the purpose of retrieved context

   Retrieved context provides additional information that the model might not have seen during training.

2. Step 2: Connect context to model output quality

   Providing this extra information helps the model give better and more accurate answers.

3. Final Answer:

   To give the model extra information it did not learn before -> Option A

4. Quick Check:

   Extra info improves answers = D [OK]

Hint: Extra info helps model answer better [OK]

Common Mistakes:

• Thinking context speeds up the model
• Believing context shrinks the model size
• Assuming context replaces training data

2. Which of the following is the correct way to combine retrieved context with an LLM prompt?

easy

A. prompt = question * context

B. prompt = question + context

C. prompt = context + ' ' + question

D. prompt = context - question

Solution

1. Step 1: Understand prompt construction

   The prompt should start with the context followed by the question to give the model relevant info first.

2. Step 2: Check syntax correctness

   Using string concatenation with '+' is correct; multiplication or subtraction of strings is invalid.

3. Final Answer:

   prompt = context + ' ' + question -> Option C

4. Quick Check:

   Context before question with '+' = A [OK]

Hint: Concatenate context and question with + [OK]

Common Mistakes:

• Putting question before context
• Using * or - operators on strings
• Not adding space between context and question

3. Given the code below, what will be the output?

context = 'The capital of France is Paris.'
question = 'What is the capital of France?'
prompt = context + ' ' + question
response = llm.generate(prompt)
print(response)

Assuming llm.generate() returns the model's answer, what is the likely output?

medium

A. Paris

B. London

C. I don't know

D. Error: undefined variable

Solution

1. Step 1: Analyze the prompt content

   The prompt includes the context 'The capital of France is Paris.' followed by the question.

2. Step 2: Predict model output based on context

   The model uses the context to answer correctly with 'Paris'.

3. Final Answer:

   Paris -> Option A

4. Quick Check:

   Context guides answer = Paris [OK]

Hint: Context gives correct answer to question [OK]

Common Mistakes:

• Ignoring context and guessing wrong
• Assuming code error without cause
• Thinking model says 'I don't know'

4. You wrote this code to combine context with a question:

context = 'Water boils at 100 degrees Celsius.'
question = 'At what temperature does water boil?'
prompt = question + ' ' + context
response = llm.generate(prompt)
print(response)

Why might the model give a less accurate answer?

medium

A. Because the context is missing important info

B. Because the question comes before the context, confusing the model

C. Because the model cannot handle string concatenation

D. Because the prompt is too short

Solution

1. Step 1: Check prompt order

   The prompt puts the question before the context, which may confuse the model about what info to use.

2. Step 2: Understand best practice

   Context should come first to provide relevant info before the question.

3. Final Answer:

   Because the question comes before the context, confusing the model -> Option B

4. Quick Check:

   Context before question improves accuracy = B [OK]

Hint: Put context before question in prompt [OK]

Common Mistakes:

• Thinking model can't concatenate strings
• Assuming context lacks info
• Believing prompt length is the issue

5. You want to build a system that answers questions about a company's products using an LLM. You have a large product manual. What is the best way to combine the manual with the LLM to get accurate answers?

hard

A. Train a new LLM from scratch on the manual

B. Feed the entire manual as a prompt to the LLM every time

C. Only ask the question without any manual context

D. Retrieve relevant sections from the manual and add them as context before the question in the prompt

Solution

1. Step 1: Consider prompt size limits

   Feeding the entire manual is too large and inefficient for the LLM prompt.

2. Step 2: Use retrieval to select relevant info

   Retrieving relevant sections and adding them as context helps the model answer accurately without overload.

3. Step 3: Evaluate other options

   Asking without context misses info; training new LLM is costly and unnecessary.

4. Final Answer:

   Retrieve relevant sections from the manual and add them as context before the question in the prompt -> Option D

5. Quick Check:

   Relevant context retrieval + LLM = A [OK]

Hint: Retrieve relevant info, then prompt LLM [OK]

Common Mistakes:

• Trying to input entire manual at once
• Ignoring context and asking only question
• Thinking retraining is always needed