Multimodal RAG in Prompt Engineering / GenAI - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to load a pretrained multimodal retriever model.

Prompt Engineering / GenAI

from transformers import [1]
retriever = [1].from_pretrained('multimodal-rag-base')

Drag options to blanks, or click blank then click option'

ARagSequenceForGeneration

BRagTokenizer

CRagRetriever

DRagConfig

Attempts:

3 left

💡 Hint

Common Mistakes

Using RagTokenizer instead of RagRetriever.

Using RagConfig which is only for configuration.

The RagRetriever class is used to load a pretrained retriever model for RAG architectures.

2fill in blank

medium

Complete the code to encode an image input for the multimodal retriever.

Prompt Engineering / GenAI

from PIL import Image
image = Image.open('input.jpg')
inputs = retriever.image_encoder.[1](image, return_tensors='pt')

Drag options to blanks, or click blank then click option'

Aforward

Bencode

Cprocess

Dencode_image

Attempts:

3 left

💡 Hint

Common Mistakes

Using encode which is generic and may not exist.

Using forward directly without preprocessing.

The method encode_image is used to convert images into embeddings for retrieval.

3fill in blank

hard

Fix the error in the code to retrieve documents using the multimodal retriever.

Prompt Engineering / GenAI

retrieved_docs = retriever.[1](inputs['pixel_values'], top_k=5)

Drag options to blanks, or click blank then click option'

Aget_relevant_documents

Bretrieve_docs

Cretrieve_documents

Dretrieve

Attempts:

3 left

💡 Hint

Common Mistakes

Using retrieve which is not a method.

Using retrieve_docs which does not exist.

The correct method to get relevant documents from the retriever is get_relevant_documents.

4fill in blank

hard

Fill both blanks to create a multimodal RAG model and generate an answer.

Prompt Engineering / GenAI

from transformers import RagSequenceForGeneration
model = RagSequenceForGeneration.from_pretrained('multimodal-rag-base')
outputs = model.generate([1], [2]=retrieved_docs)

Drag options to blanks, or click blank then click option'

Ainput_ids

Bcontext_input_ids

Cretrieved_docs

Dcontext_docs

Attempts:

3 left

💡 Hint

Common Mistakes

Passing retrieved_docs as a positional argument instead of context_input_ids.

Using wrong keyword arguments.

The generate method requires input_ids and context_input_ids to provide the question and retrieved documents.

5fill in blank

hard

Fill all three blanks to prepare inputs, retrieve documents, and generate answers in a multimodal RAG pipeline.

Prompt Engineering / GenAI

question = 'What is shown in the image?'
inputs = retriever.question_encoder.tokenizer(question, return_tensors='pt')
image_inputs = retriever.image_encoder.[1](image, return_tensors='pt')
retrieved_docs = retriever.get_relevant_documents([2])
outputs = model.generate(input_ids=inputs['input_ids'], [3]=retrieved_docs)

Drag options to blanks, or click blank then click option'

Aencode_image

Bimage_inputs['pixel_values']

Ccontext_input_ids

Dinput_ids

Attempts:

3 left

💡 Hint

Common Mistakes

Passing the whole image_inputs dict instead of pixel_values.

Using input_ids instead of context_input_ids for retrieved docs.

First, encode the image with encode_image, then pass pixel_values to get_relevant_documents, and finally use context_input_ids to generate answers.

Practice

1. What is the main purpose of Multimodal RAG in AI systems?

easy

A. To generate images from text descriptions without retrieval

B. To translate languages using only text data

C. To combine text and images for better information retrieval and generation

D. To classify images into categories without text input

Solution

1. Step 1: Understand the components of Multimodal RAG

   Multimodal RAG uses both text and image data to improve retrieval and generation tasks.

2. Step 2: Identify the main goal

   The goal is to combine these data types to find and generate better answers than using text or images alone.

3. Final Answer:

   To combine text and images for better information retrieval and generation -> Option C

4. Quick Check:

   Multimodal RAG = combine text + images [OK]

Hint: Remember: Multimodal means multiple data types combined [OK]

Common Mistakes:

• Thinking it only works with text
• Confusing it with image-only models
• Assuming it only generates images

2. Which of the following is the correct component setup for a Multimodal RAG system?

easy

A. Single encoder for both text and images, no retriever

B. Separate encoders for text and images, plus a retriever and a generator

C. Only a text encoder and a generator, no image processing

D. Only an image encoder and a retriever, no text input

Solution

1. Step 1: Recall the architecture of Multimodal RAG

   It uses separate encoders for text and images to handle each data type properly.

2. Step 2: Understand the role of retriever and generator

   The retriever finds relevant data, and the generator creates the final output combining both modalities.

3. Final Answer:

   Separate encoders for text and images, plus a retriever and a generator -> Option B

4. Quick Check:

   Separate encoders + retriever + generator = B [OK]

Hint: Look for separate encoders and both retriever and generator [OK]

Common Mistakes:

• Assuming one encoder handles both text and images
• Ignoring the retriever component
• Thinking image processing is optional

3. Given the following pseudocode for a Multimodal RAG retrieval step, what will be the output type?

text_embedding = text_encoder(text_input)
image_embedding = image_encoder(image_input)
combined_embedding = concatenate(text_embedding, image_embedding)
retrieved_docs = retriever.retrieve(combined_embedding)
print(type(retrieved_docs))

medium

A. <class 'int'>

B. <class 'dict'>

C. <class 'str'>

D. <class 'list'>

Solution

1. Step 1: Understand the retriever output

   The retriever typically returns a list of documents or data items relevant to the query embedding.

2. Step 2: Identify the output type printed

   Since retrieved_docs holds multiple documents, its type is a list.

3. Final Answer:

   <class 'list'> -> Option D

4. Quick Check:

   Retriever output = list of documents [OK]

Hint: Retriever returns a list of relevant documents [OK]

Common Mistakes:

• Assuming output is a string or dictionary
• Confusing embedding types with retrieval output
• Expecting a single document instead of a list

4. You have this code snippet for a Multimodal RAG generator:

def generate_answer(text, image):
    text_emb = text_encoder(text)
    image_emb = image_encoder(image)
    combined = text_emb + image_emb
    docs = retriever.retrieve(combined)
    answer = generator.generate(docs)
    return answer

What is the main error in this code?

medium

A. Using '+' to combine embeddings instead of concatenation

B. Missing image encoder call

C. Retriever should not be called before generator

D. Generator cannot take documents as input

Solution

1. Step 1: Check how embeddings are combined

   Embeddings from different modalities should be concatenated, not added, to preserve information.

2. Step 2: Understand the impact of using '+' operator

   Adding embeddings sums values element-wise, which can lose modality-specific features.

3. Final Answer:

   Using '+' to combine embeddings instead of concatenation -> Option A

4. Quick Check:

   Combine embeddings = concatenate, not add [OK]

Hint: Use concatenate, not plus, to combine embeddings [OK]

Common Mistakes:

• Thinking '+' merges embeddings correctly
• Ignoring the need for separate encoders
• Assuming retriever or generator order is wrong

5. You want to improve a Multimodal RAG system that sometimes misses relevant images when answering questions. Which approach is best to fix this?

hard

A. Train the image encoder with more diverse image-text pairs to improve embedding quality

B. Remove the retriever and rely only on the generator

C. Use only text data and ignore images to simplify the model

D. Replace the text encoder with a simpler model to speed up processing

Solution

1. Step 1: Identify the cause of missing relevant images

   Low-quality image embeddings can cause the retriever to miss relevant images.

2. Step 2: Choose the best fix

   Training the image encoder with more diverse data improves embedding quality and retrieval accuracy.

3. Final Answer:

   Train the image encoder with more diverse image-text pairs to improve embedding quality -> Option A

4. Quick Check:

   Better image encoder training = better retrieval [OK]

Hint: Improve encoder training with diverse data for better retrieval [OK]

Common Mistakes:

• Removing retriever loses retrieval benefits
• Ignoring images reduces multimodal power
• Simplifying text encoder won't fix image retrieval