Image understanding and description in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Image understanding and description

Which metric matters for Image understanding and description and WHY

For image understanding and description, we want to check how well the model describes images in words. Common metrics are BLEU, METEOR, ROUGE, and CIDEr. These compare the model's description to human-written ones. They matter because they tell us if the model's words match what a person would say about the image.

Also, accuracy-like metrics on object detection or classification parts help check if the model sees the right things in the image. But for description, language similarity scores are key.

Confusion matrix or equivalent visualization

Image description is not a simple yes/no task, so confusion matrix is less common. But for object detection inside the image, confusion matrix can show:

      | Predicted Object | Actual Object |
      |------------------|---------------|
      | True Positive (TP) | Correctly detected object |
      | False Positive (FP) | Wrongly detected object |
      | False Negative (FN) | Missed object |
      | True Negative (TN) | Correctly ignored background |
    

For description, we use scores like BLEU that count matching words or phrases instead.

Precision vs Recall tradeoff with concrete examples

In object detection part of image understanding:

• Precision means: When the model says "there is a cat," how often is it right? High precision means few false alarms.
• Recall means: Of all cats in the image, how many did the model find? High recall means it misses few cats.

Tradeoff example: A model that finds every cat (high recall) but sometimes says "cat" when there is none (low precision) can annoy users with wrong info.

For description, the tradeoff is between being detailed (high recall of image content) and being accurate (high precision in words). Too much detail can confuse, too little misses info.

What "good" vs "bad" metric values look like for this use case

Good values:

• BLEU score above 0.5 means the description shares many words with human captions.
• Precision and recall above 0.7 for detected objects means the model sees most objects correctly.
• CIDEr score above 1.0 shows good consensus with human descriptions.

Bad values:

• BLEU below 0.2 means the description is very different from human captions.
• Precision or recall below 0.4 means many mistakes or misses in object detection.
• Low CIDEr means poor quality or irrelevant descriptions.

Metrics pitfalls

• Accuracy paradox: A model might describe common objects well but fail on rare ones, inflating scores.
• Data leakage: If test images or captions were seen during training, scores look better but don't reflect real ability.
• Overfitting: Model memorizes training captions and scores high on training but low on new images.
• Metric limits: BLEU and similar scores don't capture creativity or meaning fully, so human review is important.

Self-check question

Your image description model has 98% accuracy on object detection but only 12% recall on rare objects. Is it good for production? Why not?

Answer: No, because the model misses most rare objects (low recall). This means it often fails to describe important parts of images. High accuracy alone is misleading if it mostly sees common objects.

Key Result

For image understanding and description, language similarity scores like BLEU and CIDEr combined with precision and recall on object detection best show model quality.