CLIP (vision-language model) in Computer Vision - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

How does CLIP learn to connect images and text?

CLIP is a model that understands images and text together. How does it learn this connection?

A By training only on images to classify objects without any text input.

B By clustering images based on color patterns without using text.

C By generating text captions from images using a language-only model.

D By training on pairs of images and their matching text descriptions to align their representations.

💡 Hint

Think about how the model sees both images and text at the same time during training.

CLIP learns by looking at many image-text pairs and training its image and text parts to produce similar outputs for matching pairs. This way, it connects images and their descriptions.

❓ Predict Output

intermediate

What is the output shape of CLIP's image encoder?

Given a batch of 8 images, each resized to 224x224 pixels with 3 color channels, what is the shape of the output from CLIP's image encoder?

Computer Vision

import torch
from torchvision.transforms import Compose, Resize, ToTensor, Normalize

# Dummy batch of 8 images
images = torch.randn(8, 3, 224, 224)

# Assume clip_model is loaded and has an image encoder
# output = clip_model.encode_image(images)

# What is output.shape?

A (8, 3, 224, 224)

B (8, 512)

C (8, 1024)

D (224, 224, 3)

💡 Hint

The image encoder outputs a vector embedding per image, not an image tensor.

CLIP's image encoder outputs a fixed-length vector (usually 512 dimensions) for each image in the batch. So for 8 images, the output shape is (8, 512).

❓ Model Choice

advanced

Which architecture is used for CLIP's text encoder?

CLIP uses a specific type of neural network to process text input. Which one is it?

A A transformer-based model that processes sequences of words.

B A convolutional neural network (CNN) designed for images.

C A recurrent neural network (RNN) with LSTM units.

D A simple feedforward neural network with no sequence handling.

💡 Hint

Think about models good at understanding sequences and context in language.

CLIP's text encoder is a transformer model, which is excellent at handling sequences of words and capturing context.

❓ Metrics

advanced

Which metric best evaluates CLIP's zero-shot classification accuracy?

CLIP can classify images without training on specific classes. Which metric measures how well it does this?

A Top-1 accuracy comparing predicted labels to true labels.

B BLEU score measuring text generation quality.

C Mean Squared Error (MSE) between image pixels and text tokens.

D Perplexity of the language model on text input.

💡 Hint

Think about how classification models are usually evaluated.

Top-1 accuracy measures how often the model's top predicted class matches the true class, which is suitable for zero-shot classification evaluation.

🔧 Debug

expert

Why does CLIP's similarity score between image and text embeddings sometimes produce negative values?

CLIP computes similarity scores between image and text vectors. Sometimes these scores are negative. Why?

A Because the image encoder outputs random noise vectors.

B Because the model outputs raw logits that are always negative.

C Because the embeddings are normalized and cosine similarity can range from -1 to 1.

D Because the text encoder uses ReLU activations that produce negative values.

💡 Hint

Recall the range of cosine similarity values between two vectors.

CLIP uses cosine similarity between normalized embeddings, which ranges from -1 (opposite) to 1 (identical). Negative values mean the vectors point in opposite directions.