Vision Transformer (ViT) in Computer Vision - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of the patch embedding step in a Vision Transformer?

In Vision Transformers, images are split into patches before processing. What is the main reason for this patch embedding step?

A To perform data augmentation by randomly cropping image patches

B To convert the image into a sequence of tokens suitable for transformer input

C To reduce the image resolution to speed up convolution operations

D To apply convolutional filters to extract local features

💡 Hint

Think about how transformers process data compared to convolutional neural networks.

Vision Transformers split images into patches and flatten them to create a sequence of tokens. This sequence format is required because transformers are designed to process sequential data, like words in a sentence.

❓ Predict Output

intermediate

Output shape after patch embedding in ViT

Given a batch of images with shape (batch_size=8, height=32, width=32, channels=3), and a patch size of 8, what is the shape of the patch embeddings after flattening and linear projection?

Computer Vision

import torch
import torch.nn as nn

batch_size = 8
img_size = 32
patch_size = 8
channels = 3

images = torch.randn(batch_size, channels, img_size, img_size)

num_patches = (img_size // patch_size) ** 2
patch_dim = channels * patch_size * patch_size

patches = images.unfold(2, patch_size, patch_size).unfold(3, patch_size, patch_size)
patches = patches.contiguous().view(batch_size, channels, -1, patch_size, patch_size)
patches = patches.permute(0, 2, 1, 3, 4).contiguous().view(batch_size, num_patches, patch_dim)

linear_proj = nn.Linear(patch_dim, 64)
patch_embeddings = linear_proj(patches)

output_shape = patch_embeddings.shape

A (8, 64, 16)

B (8, 16, 192)

C (8, 16, 64)

D (8, 32, 64)

💡 Hint

Calculate how many patches fit in the image and the embedding dimension after projection.

The image is split into (32/8)^2 = 16 patches. Each patch is flattened to size 3*8*8=192. The linear layer projects this to 64 dimensions. So the output shape is (batch_size, 16, 64).

❓ Model Choice

advanced

Choosing the correct positional encoding for ViT

Which type of positional encoding is commonly used in Vision Transformers to help the model understand the order and position of image patches?

A Learnable positional embeddings added to patch embeddings

B Fixed sinusoidal positional encodings like in original NLP transformers

C No positional encoding is used in ViT

D Positional encoding applied via convolutional layers

💡 Hint

Consider whether the positional encoding is fixed or learned in ViT implementations.

Vision Transformers typically use learnable positional embeddings that are added to the patch embeddings. This allows the model to learn the best way to represent position information during training.

❓ Hyperparameter

advanced

Effect of increasing the number of transformer layers in ViT

What is the most likely effect of increasing the number of transformer encoder layers in a Vision Transformer model?

A Improves model capacity and may increase accuracy but also increases training time and risk of overfitting

B Decreases model capacity and reduces accuracy due to vanishing gradients

C Has no effect on model performance or training time

D Reduces the number of patches processed by the model

💡 Hint

Think about how deeper models affect learning and computation.

Adding more transformer layers increases the model's ability to learn complex patterns but also requires more computation and can lead to overfitting if not managed properly.

❓ Metrics

expert

Interpreting ViT training loss and accuracy curves

During training of a Vision Transformer on image classification, the training loss steadily decreases but the validation accuracy plateaus early and does not improve. What is the most likely explanation?

A The batch size is too large causing poor gradient estimates

B The model is underfitting and needs more training epochs

C The learning rate is too high causing unstable training

D The model is overfitting the training data and not generalizing well to validation data

💡 Hint

Consider what it means when training loss improves but validation accuracy stops improving.

If training loss decreases but validation accuracy stops improving, the model is likely memorizing training data patterns but failing to generalize, indicating overfitting.