Vision-language models (GPT-4V) in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding the core capability of GPT-4V

What is the primary advantage of GPT-4V compared to traditional language-only models?

A It focuses solely on generating images from text prompts.

B It replaces the need for any text input by using images alone.

C It can process and understand both images and text inputs simultaneously.

D It only improves text generation speed without image understanding.

💡 Hint

Think about what 'vision-language' means in the model's name.

GPT-4V combines visual and textual understanding, allowing it to interpret images and text together, unlike traditional models that only handle text.

❓ Model Choice

intermediate

Choosing the right model for multimodal tasks

You want to build an application that answers questions about photos users upload. Which model is best suited for this task?

A GPT-3, because it is the most advanced text-only model.

B GPT-4V, because it can understand images and text together.

C A convolutional neural network (CNN) trained only on image classification.

D A text summarization model trained on news articles.

💡 Hint

Consider which model can handle both images and text inputs.

GPT-4V is designed to understand images and text jointly, making it ideal for answering questions about photos, unlike text-only or image-only models.

❓ Predict Output

advanced

Predicting GPT-4V output for image-text input

Given an image of a red apple and the text prompt 'What fruit is this?', what is the most likely output from GPT-4V?

A "This is a red apple."

B "This is a green apple."

C "This is a banana."

D "I cannot see any fruit in the image."

💡 Hint

GPT-4V identifies objects in images and answers questions about them.

GPT-4V uses its vision-language ability to recognize the red apple in the image and respond accurately to the question.

❓ Metrics

advanced

Evaluating GPT-4V's performance on multimodal tasks

Which metric best measures GPT-4V's accuracy in answering questions about images?

A Perplexity, which measures how well the model predicts the next word in text.

B BLEU score, which measures similarity between generated and reference text only.

C Image classification accuracy, which only measures image label correctness without text.

D Multimodal accuracy, which checks if the model's answer matches the correct text for the given image and question.

💡 Hint

Think about a metric that evaluates both image understanding and text generation.

Multimodal accuracy evaluates if the model correctly answers questions based on both image content and text input, which is essential for GPT-4V.

🔧 Debug

expert

Diagnosing GPT-4V's incorrect image question answering

You notice GPT-4V often answers incorrectly when images contain multiple objects. What is the most likely cause?

A The model's attention mechanism may not effectively focus on the relevant object in complex scenes.

B The model cannot process images larger than 64x64 pixels.

C The text input is ignored when multiple objects are present in the image.

D The model only works with black and white images, so color images cause errors.

💡 Hint

Consider how the model handles complex visual scenes with many details.

GPT-4V uses attention to link text and image parts; if it struggles to focus on the right object among many, answers can be wrong.