Vision Transformer (ViT) in Computer Vision - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of the patch embedding step in a Vision Transformer?

In Vision Transformers, images are split into patches before processing. What is the main reason for this patch embedding step?

A To perform data augmentation by randomly cropping image patches

B To convert the image into a sequence of tokens suitable for transformer input

C To reduce the image resolution to speed up convolution operations

D To apply convolutional filters to extract local features

💡 Hint

Think about how transformers process data compared to convolutional neural networks.

Vision Transformers split images into patches and flatten them to create a sequence of tokens. This sequence format is required because transformers are designed to process sequential data, like words in a sentence.

❓ Predict Output

intermediate

Output shape after patch embedding in ViT

Given a batch of images with shape (batch_size=8, height=32, width=32, channels=3), and a patch size of 8, what is the shape of the patch embeddings after flattening and linear projection?

Computer Vision

import torch
import torch.nn as nn

batch_size = 8
img_size = 32
patch_size = 8
channels = 3

images = torch.randn(batch_size, channels, img_size, img_size)

num_patches = (img_size // patch_size) ** 2
patch_dim = channels * patch_size * patch_size

patches = images.unfold(2, patch_size, patch_size).unfold(3, patch_size, patch_size)
patches = patches.contiguous().view(batch_size, channels, -1, patch_size, patch_size)
patches = patches.permute(0, 2, 1, 3, 4).contiguous().view(batch_size, num_patches, patch_dim)

linear_proj = nn.Linear(patch_dim, 64)
patch_embeddings = linear_proj(patches)

output_shape = patch_embeddings.shape

A (8, 64, 16)

B (8, 16, 192)

C (8, 16, 64)

D (8, 32, 64)

💡 Hint

Calculate how many patches fit in the image and the embedding dimension after projection.

The image is split into (32/8)^2 = 16 patches. Each patch is flattened to size 3*8*8=192. The linear layer projects this to 64 dimensions. So the output shape is (batch_size, 16, 64).

❓ Model Choice

advanced

Choosing the correct positional encoding for ViT

Which type of positional encoding is commonly used in Vision Transformers to help the model understand the order and position of image patches?

A Learnable positional embeddings added to patch embeddings

B Fixed sinusoidal positional encodings like in original NLP transformers

C No positional encoding is used in ViT

D Positional encoding applied via convolutional layers

💡 Hint

Consider whether the positional encoding is fixed or learned in ViT implementations.

Vision Transformers typically use learnable positional embeddings that are added to the patch embeddings. This allows the model to learn the best way to represent position information during training.

❓ Hyperparameter

advanced

Effect of increasing the number of transformer layers in ViT

What is the most likely effect of increasing the number of transformer encoder layers in a Vision Transformer model?

A Improves model capacity and may increase accuracy but also increases training time and risk of overfitting

B Decreases model capacity and reduces accuracy due to vanishing gradients

C Has no effect on model performance or training time

D Reduces the number of patches processed by the model

💡 Hint

Think about how deeper models affect learning and computation.

Adding more transformer layers increases the model's ability to learn complex patterns but also requires more computation and can lead to overfitting if not managed properly.

❓ Metrics

expert

Interpreting ViT training loss and accuracy curves

During training of a Vision Transformer on image classification, the training loss steadily decreases but the validation accuracy plateaus early and does not improve. What is the most likely explanation?

A The batch size is too large causing poor gradient estimates

B The model is underfitting and needs more training epochs

C The learning rate is too high causing unstable training

D The model is overfitting the training data and not generalizing well to validation data

💡 Hint

Consider what it means when training loss improves but validation accuracy stops improving.

If training loss decreases but validation accuracy stops improving, the model is likely memorizing training data patterns but failing to generalize, indicating overfitting.

Practice

1. What is the main purpose of splitting an image into patches in a Vision Transformer (ViT)?

easy

A. To reduce the image size by cropping

B. To convert the image into smaller parts that the transformer can process as tokens

C. To apply convolution filters on each patch separately

D. To increase the image resolution for better detail

Solution

1. Step 1: Understand ViT input processing

   ViT splits images into fixed-size patches to treat each patch like a word token in language models.

2. Step 2: Purpose of patch splitting

   This allows the transformer to process image patches as a sequence, enabling attention mechanisms to learn relationships.

3. Final Answer:

   To convert the image into smaller parts that the transformer can process as tokens -> Option B

4. Quick Check:

   Image patches = tokens for transformer [OK]

Hint: Think of patches as words in a sentence for the transformer [OK]

Common Mistakes:

• Confusing patch splitting with image resizing
• Thinking patches are processed by convolution
• Assuming patches increase image resolution

2. Which of the following is the correct way to add a class token to the patch embeddings in ViT using Python-like pseudocode?

easy

A. patches = torch.cat([class_token, patches], dim=1)

B. patches = torch.cat([patches, class_token], dim=1)

C. patches = torch.cat([patches, class_token], dim=0)

D. patches = torch.cat([class_token, patches], dim=0)

Solution

1. Step 1: Understand tensor concatenation dimension

   Patch embeddings are sequences along dimension 1 (batch, seq, embed); class token must be prepended along this dimension.

2. Step 2: Correct concatenation syntax

   Using torch.cat with dim=1 adds class_token at the start of the sequence correctly.

3. Final Answer:

   patches = torch.cat([class_token, patches], dim=1) -> Option A

4. Quick Check:

   Class token prepended along sequence dim = patches = torch.cat([class_token, patches], dim=1) [OK]

Hint: Class token goes first, concat along sequence dimension (dim=1) [OK]

Common Mistakes:

• Concatenating along wrong dimension (dim=0)
• Appending class token at the end instead of start
• Mixing order of tensors in concat

3. Given the following simplified ViT patch embedding code, what is the shape of patch_embeddings after processing a batch of 8 images of size 32x32 with patch size 8 and embedding dimension 64?

patch_size = 8
embedding_dim = 64
batch_size = 8
image_size = 32
num_patches = (image_size // patch_size) ** 2
patch_embeddings = torch.randn(batch_size, num_patches, embedding_dim)

medium

A. (16, 8, 64)

B. (8, 64, 16)

C. (8, 8, 64)

D. (8, 16, 64)

Solution

1. Step 1: Calculate number of patches

   Number of patches = (32 / 8)^2 = 4^2 = 16 patches per image.

2. Step 2: Determine patch_embeddings shape

   Shape is (batch_size, num_patches, embedding_dim) = (8, 16, 64).

3. Final Answer:

   (8, 16, 64) -> Option D

4. Quick Check:

   Batch=8, patches=16, embed=64 [OK]

Hint: Calculate patches as (image/patch)^2, then batch x patches x embed [OK]

Common Mistakes:

• Mixing embedding dimension and patch count order
• Calculating patches incorrectly
• Confusing batch size with patch count

4. You have this ViT code snippet that throws an error:

class_token = torch.randn(1, 1, 64)
patches = torch.randn(8, 16, 64)
input_seq = torch.cat([class_token, patches], dim=1)

What is the cause of the error?

medium

A. Embedding dimensions do not match

B. Wrong concatenation dimension; should be dim=0

C. class_token shape should be (8, 1, 64) to match batch size

D. Dimension mismatch because class_token sequence size is 1 but patches sequence size is 16

Solution

1. Step 1: Check batch size compatibility

   class_token has batch size 1, patches have batch size 8; they must match for concatenation.

2. Step 2: Fix class_token shape

   class_token should be repeated or created with shape (8, 1, 64) to match patches batch size.

3. Final Answer:

   class_token shape should be (8, 1, 64) to match batch size -> Option C

4. Quick Check:

   Batch sizes must match for concat [OK]

Hint: Match batch sizes before concatenating tensors [OK]

Common Mistakes:

• Ignoring batch size mismatch
• Changing wrong concat dimension
• Assuming embedding dims cause error

5. In a Vision Transformer model, why is the class token important for image classification tasks?

hard

A. It aggregates information from all patches via attention to produce a final image representation

B. It stores the positional information of patches

C. It applies convolution to patches before transformer layers

D. It normalizes the patch embeddings before feeding to the transformer

Solution

1. Step 1: Understand class token role

   The class token is a special token that attends to all patch tokens and gathers their information.

2. Step 2: Use in classification

   After transformer layers, the class token embedding is used as the image's summary representation for classification.

3. Final Answer:

   It aggregates information from all patches via attention to produce a final image representation -> Option A

4. Quick Check:

   Class token = image summary for classification [OK]

Hint: Class token collects info from patches for final decision [OK]

Common Mistakes:

• Confusing class token with positional encoding
• Thinking class token applies convolution
• Assuming class token normalizes embeddings