Combining retrieved context with LLM in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

How does retrieved context improve LLM responses?

Imagine you ask a large language model (LLM) a question. How does adding retrieved context from a database help the LLM give better answers?

A It reduces the LLM's vocabulary size to speed up response time.

B It provides extra relevant information so the LLM can generate more accurate and specific responses.

C It replaces the LLM's internal knowledge completely with the retrieved data.

D It forces the LLM to ignore the question and only use the retrieved context.

💡 Hint

Think about how having more facts related to your question helps you answer better.

Retrieved context adds relevant facts or documents that the LLM can use to generate more accurate and detailed answers. It supplements the LLM's internal knowledge without replacing it.

❓ Predict Output

intermediate

Output of combining retrieved context with LLM prompt

Given the following Python code snippet that combines retrieved context with a prompt for an LLM, what is the printed output?

Prompt Engineering / GenAI

retrieved_context = "The Eiffel Tower is in Paris."
prompt = "Where is the Eiffel Tower located?"
combined_input = f"Context: {retrieved_context}\nQuestion: {prompt}"
print(combined_input)

A

Context: The Eiffel Tower is in Paris.
Question: Where is the Eiffel Tower located?

B Context: The Eiffel Tower is in Paris. Question: Where is the Eiffel Tower located?

C

Context: retrieved_context
Question: prompt

D The Eiffel Tower is in Paris. Where is the Eiffel Tower located?

💡 Hint

Look carefully at how the f-string formats the variables with newlines.

The f-string inserts the retrieved context and prompt with a newline between them, so the output shows 'Context: ...' on one line and 'Question: ...' on the next.

❓ Model Choice

advanced

Best model architecture for combining retrieved context with LLM

Which model architecture is best suited to effectively combine retrieved context with a large language model for question answering?

A A recurrent neural network that processes only the retrieved context without the question.

B A simple feedforward neural network that ignores context and only processes the question.

C A convolutional neural network trained on images unrelated to text.

D A retrieval-augmented transformer that encodes context and question jointly before decoding.

💡 Hint

Think about architectures designed to handle both context and question together.

Retrieval-augmented transformers are designed to jointly encode retrieved context and the question, enabling the model to generate answers based on both inputs effectively.

❓ Hyperparameter

advanced

Choosing the number of retrieved documents for context

When combining retrieved context with an LLM, what is a key consideration when choosing how many documents to retrieve?

A Retrieving documents unrelated to the question improves diversity and accuracy.

B Retrieving only one document always guarantees the best answer.

C Retrieving too many documents can overwhelm the model and reduce answer quality due to noise.

D Retrieving zero documents is best because the LLM knows everything already.

💡 Hint

More context is not always better; think about information overload.

Retrieving too many documents can introduce irrelevant information, confusing the model and lowering answer quality. A balanced number of relevant documents is ideal.

❓ Metrics

expert

Evaluating combined retrieval and LLM system performance

You have a system that retrieves documents and then uses an LLM to answer questions. Which metric best measures how well the combined system answers questions accurately?

A Exact Match (EM) score comparing generated answers to ground truth answers.

B Mean Squared Error (MSE) between retrieved document vectors.

C BLEU score comparing retrieved documents to questions.

D Training loss of the LLM on unrelated text data.

💡 Hint

Think about metrics that compare generated answers to correct answers.

Exact Match (EM) measures the percentage of generated answers that exactly match the correct answers, making it a good metric for question answering accuracy.

Practice

1. Why do we combine retrieved context with a large language model (LLM)?

easy

A. To give the model extra information it did not learn before

B. To make the model run faster

C. To reduce the size of the model

D. To replace the model's training data

Solution

1. Step 1: Understand the purpose of retrieved context

   Retrieved context provides additional information that the model might not have seen during training.

2. Step 2: Connect context to model output quality

   Providing this extra information helps the model give better and more accurate answers.

3. Final Answer:

   To give the model extra information it did not learn before -> Option A

4. Quick Check:

   Extra info improves answers = D [OK]

Hint: Extra info helps model answer better [OK]

Common Mistakes:

• Thinking context speeds up the model
• Believing context shrinks the model size
• Assuming context replaces training data

2. Which of the following is the correct way to combine retrieved context with an LLM prompt?

easy

A. prompt = question * context

B. prompt = question + context

C. prompt = context + ' ' + question

D. prompt = context - question

Solution

1. Step 1: Understand prompt construction

   The prompt should start with the context followed by the question to give the model relevant info first.

2. Step 2: Check syntax correctness

   Using string concatenation with '+' is correct; multiplication or subtraction of strings is invalid.

3. Final Answer:

   prompt = context + ' ' + question -> Option C

4. Quick Check:

   Context before question with '+' = A [OK]

Hint: Concatenate context and question with + [OK]

Common Mistakes:

• Putting question before context
• Using * or - operators on strings
• Not adding space between context and question

3. Given the code below, what will be the output?

context = 'The capital of France is Paris.'
question = 'What is the capital of France?'
prompt = context + ' ' + question
response = llm.generate(prompt)
print(response)

Assuming llm.generate() returns the model's answer, what is the likely output?

medium

A. Paris

B. London

C. I don't know

D. Error: undefined variable

Solution

1. Step 1: Analyze the prompt content

   The prompt includes the context 'The capital of France is Paris.' followed by the question.

2. Step 2: Predict model output based on context

   The model uses the context to answer correctly with 'Paris'.

3. Final Answer:

   Paris -> Option A

4. Quick Check:

   Context guides answer = Paris [OK]

Hint: Context gives correct answer to question [OK]

Common Mistakes:

• Ignoring context and guessing wrong
• Assuming code error without cause
• Thinking model says 'I don't know'

4. You wrote this code to combine context with a question:

context = 'Water boils at 100 degrees Celsius.'
question = 'At what temperature does water boil?'
prompt = question + ' ' + context
response = llm.generate(prompt)
print(response)

Why might the model give a less accurate answer?

medium

A. Because the context is missing important info

B. Because the question comes before the context, confusing the model

C. Because the model cannot handle string concatenation

D. Because the prompt is too short

Solution

1. Step 1: Check prompt order

   The prompt puts the question before the context, which may confuse the model about what info to use.

2. Step 2: Understand best practice

   Context should come first to provide relevant info before the question.

3. Final Answer:

   Because the question comes before the context, confusing the model -> Option B

4. Quick Check:

   Context before question improves accuracy = B [OK]

Hint: Put context before question in prompt [OK]

Common Mistakes:

• Thinking model can't concatenate strings
• Assuming context lacks info
• Believing prompt length is the issue

5. You want to build a system that answers questions about a company's products using an LLM. You have a large product manual. What is the best way to combine the manual with the LLM to get accurate answers?

hard

A. Train a new LLM from scratch on the manual

B. Feed the entire manual as a prompt to the LLM every time

C. Only ask the question without any manual context

D. Retrieve relevant sections from the manual and add them as context before the question in the prompt

Solution

1. Step 1: Consider prompt size limits

   Feeding the entire manual is too large and inefficient for the LLM prompt.

2. Step 2: Use retrieval to select relevant info

   Retrieving relevant sections and adding them as context helps the model answer accurately without overload.

3. Step 3: Evaluate other options

   Asking without context misses info; training new LLM is costly and unnecessary.

4. Final Answer:

   Retrieve relevant sections from the manual and add them as context before the question in the prompt -> Option D

5. Quick Check:

   Relevant context retrieval + LLM = A [OK]

Hint: Retrieve relevant info, then prompt LLM [OK]

Common Mistakes:

• Trying to input entire manual at once
• Ignoring context and asking only question
• Thinking retraining is always needed