Vision-language models (GPT-4V) in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Vision-language models (GPT-4V)

Which metric matters for Vision-language models (GPT-4V) and WHY

Vision-language models like GPT-4V combine images and text. We want to check how well the model understands both. Key metrics include accuracy for classification tasks, BLEU or ROUGE for text generation quality, and precision and recall when detecting objects or answering questions about images. These metrics tell us if the model gives correct answers, describes images well, and finds important details without too many mistakes.

Confusion matrix example for image question answering

      | Predicted Yes | Predicted No |
      |---------------|--------------|
      | True Positives (TP) = 80  | False Negatives (FN) = 20 |
      | False Positives (FP) = 10 | True Negatives (TN) = 90 |

      Total samples = 80 + 20 + 10 + 90 = 200

      Precision = TP / (TP + FP) = 80 / (80 + 10) = 0.89
      Recall = TP / (TP + FN) = 80 / (80 + 20) = 0.80
      Accuracy = (TP + TN) / Total = (80 + 90) / 200 = 0.85
    

This matrix helps us see where the model makes mistakes: missing true answers (FN) or giving wrong answers (FP).

Precision vs Recall tradeoff with examples

Imagine GPT-4V is used to detect objects in images and describe them:

• High precision, low recall: The model only says "cat" when very sure, so it rarely makes mistakes (few false cats), but it misses some cats in images. Good if you want to avoid wrong labels.
• High recall, low precision: The model tries to find all cats, even if unsure, so it finds most cats but sometimes calls other animals cats by mistake. Good if missing any cat is bad.

Choosing depends on the task: for safety-critical tasks, recall is more important; for user experience, precision might matter more.

What "good" vs "bad" metric values look like for GPT-4V

Good metrics mean the model understands images and text well:

• Accuracy above 85% on image classification or question answering.
• Precision and recall both above 80%, showing balanced detection and correctness.
• BLEU or ROUGE scores above 0.5 for generated captions or answers, meaning text is relevant and fluent.

Bad metrics show problems:

• Accuracy below 60%, meaning many wrong answers.
• Precision very low (<50%) means many false positives.
• Recall very low (<50%) means many missed true cases.
• Very low BLEU/ROUGE (<0.3) means poor text quality.

Common pitfalls in evaluating vision-language models

• Accuracy paradox: High accuracy can be misleading if data is unbalanced (e.g., many easy images).
• Data leakage: If test images or captions appear in training, metrics look better but model won't generalize.
• Overfitting: Model performs well on training but poorly on new images, showing metrics that don't reflect real use.
• Ignoring metric tradeoffs: Focusing only on accuracy without precision/recall can hide important errors.
• Using wrong metrics: BLEU or ROUGE are for text quality, not classification accuracy.

Self-check question

Your GPT-4V model has 98% accuracy but only 12% recall on detecting rare objects in images. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is misleading because the rare objects are few, so the model mostly guesses "no" and is right. But the very low recall means it misses almost all rare objects, which is bad if detecting them is important. You need to improve recall to catch more true cases.

Key Result

Balanced precision and recall above 80% with good text quality scores indicate a well-performing vision-language model.