Embedding dimensionality considerations in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why choose higher embedding dimensions?

When creating embeddings for text data, why might you choose a higher dimensionality?

A To capture more detailed features and subtle differences between items

B To reduce the training time and computational cost

C Because higher dimensions always guarantee better accuracy regardless of data

D To make the embeddings easier to visualize in 2D or 3D plots

💡 Hint

Think about what more dimensions allow the model to represent.

Higher embedding dimensions allow the model to capture more detailed and subtle features of the data, improving representation quality. However, it may increase computational cost and risk overfitting.

❓ Metrics

intermediate

Effect of embedding size on cosine similarity

You have two sets of embeddings: one with 50 dimensions and one with 300 dimensions. Both sets are normalized. Which statement about cosine similarity between embeddings is true?

A Higher dimensional embeddings tend to have cosine similarities closer to zero due to sparsity

B Cosine similarity values are directly comparable regardless of embedding size

C Lower dimensional embeddings always produce higher cosine similarity values

D Cosine similarity cannot be computed for embeddings with different dimensions

💡 Hint

Consider how dimensionality affects vector distribution and sparsity.

Higher dimensional embeddings often become sparser, causing cosine similarity values to cluster closer to zero, making distinctions less pronounced.

❓ Predict Output

advanced

Output shape of embedding layer

Consider the following PyTorch code snippet creating an embedding layer and passing input indices:

Prompt Engineering / GenAI

import torch
import torch.nn as nn

embedding = nn.Embedding(num_embeddings=1000, embedding_dim=128)
input_indices = torch.tensor([[1, 2, 3], [4, 5, 6]])
output = embedding(input_indices)
print(output.shape)

A torch.Size([128, 3, 2])

B torch.Size([3, 2, 128])

C torch.Size([2, 128])

D torch.Size([2, 3, 128])

💡 Hint

Think about how embedding layers map indices to vectors.

The input tensor has shape (2, 3). Each index is replaced by a 128-dimensional vector, so output shape is (2, 3, 128).

❓ Hyperparameter

advanced

Choosing embedding dimension for a small dataset

You have a small dataset with only 500 unique words. Which embedding dimension is most appropriate to avoid overfitting while maintaining useful representation?

A 1024 dimensions to capture all nuances

B 3000 dimensions to ensure maximum expressiveness

C 50 dimensions to balance capacity and overfitting risk

D 5 dimensions because the dataset is small

💡 Hint

Think about the trade-off between model complexity and data size.

For small datasets, moderate embedding sizes like 50 help avoid overfitting while still capturing meaningful features.

🔧 Debug

expert

Why does increasing embedding dimension cause training to fail?

After increasing embedding dimension from 100 to 10000 in a neural network, training loss becomes NaN immediately. What is the most likely cause?

A Higher dimension embeddings always cause NaN due to numerical overflow

B The model runs out of memory causing unstable gradients

C The optimizer does not support large embedding sizes

D Embedding dimension does not affect training stability

💡 Hint

Consider hardware limits and how large parameters affect training.

Very large embeddings increase memory use and can cause unstable gradients or overflow, leading to NaN losses.

Practice

1. What does the dimensionality of an embedding vector mainly control in AI models?

easy

A. The color of the data points in visualization

B. The speed of the computer's processor

C. The level of detail or information captured about the item

D. The number of training examples needed

Solution

1. Step 1: Understand embedding vectors

   Embedding vectors represent items as numbers. Their length (dimensionality) decides how much detail they can hold.

2. Step 2: Relate dimensionality to information

   Higher dimensions mean more features can be captured, so more detail is stored about the item.

3. Final Answer:

   The level of detail or information captured about the item -> Option C

4. Quick Check:

   Embedding dimensionality = detail level [OK]

Hint: Embedding size = how detailed the vector is [OK]

Common Mistakes:

• Confusing dimensionality with training speed
• Thinking dimensionality affects data color
• Assuming dimensionality controls dataset size

2. Which of the following is the correct way to define an embedding layer with 50 dimensions in Python using PyTorch?

easy

A. nn.Embedding(dim=50, size=1000)

B. nn.Embedding(50, 1000)

C. nn.Embedding(embedding_size=50)

D. nn.Embedding(num_embeddings=1000, embedding_dim=50)

Solution

1. Step 1: Recall PyTorch embedding syntax

   PyTorch's embedding layer uses nn.Embedding(num_embeddings, embedding_dim).

2. Step 2: Match parameters to question

   We want 50 dimensions, so embedding_dim=50. Number of embeddings is usually vocabulary size, e.g., 1000.

3. Final Answer:

   nn.Embedding(num_embeddings=1000, embedding_dim=50) -> Option D

4. Quick Check:

   PyTorch embedding syntax = nn.Embedding(num_embeddings, embedding_dim) [OK]

Hint: Remember nn.Embedding(num_embeddings, embedding_dim) order [OK]

Common Mistakes:

• Swapping num_embeddings and embedding_dim
• Using wrong parameter names like dim or size
• Omitting required parameters

3. Consider this code snippet using TensorFlow to create embeddings:

embedding_layer = tf.keras.layers.Embedding(input_dim=5000, output_dim=16)
input_data = tf.constant([1, 2, 3])
output = embedding_layer(input_data)
print(output.shape)

What will be the printed shape?

medium

A. (3, 16)

B. (16, 3)

C. (3, 5000)

D. (5000, 16)

Solution

1. Step 1: Understand input and output dimensions

   Input is a list of 3 indices. Each index maps to a 16-dimensional vector.

2. Step 2: Determine output shape

   Output shape is (number of inputs, embedding dimension) = (3, 16).

3. Final Answer:

   (3, 16) -> Option A

4. Quick Check:

   Output shape = (input length, embedding dim) [OK]

Hint: Output shape = input count x embedding size [OK]

Common Mistakes:

• Confusing embedding dimension with input dimension
• Swapping rows and columns in output shape
• Assuming output shape equals input_dim

4. You have an embedding layer defined as nn.Embedding(1000, 128) in PyTorch. You try to pass an input tensor with values outside the range 0-999. What error will most likely occur?

medium

A. TypeError because input is not a float

B. IndexError due to out-of-range indices

C. ValueError because embedding dimension is wrong

D. No error, embeddings handle any input values

Solution

1. Step 1: Understand embedding input constraints

   Embedding layers expect input indices between 0 and num_embeddings-1 (0 to 999 here).

2. Step 2: Identify error from invalid indices

   Passing indices outside this range causes an IndexError because the layer cannot find embeddings for invalid indices.

3. Final Answer:

   IndexError due to out-of-range indices -> Option B

4. Quick Check:

   Embedding input indices must be valid [OK]

Hint: Embedding inputs must be valid indices [OK]

Common Mistakes:

• Thinking embeddings accept any numeric input
• Confusing input type errors with index errors
• Assuming embedding dimension affects input range

5. You want to choose the embedding dimensionality for a text classification model. The vocabulary size is 10,000 words. Which embedding size is the best balance between capturing enough detail and keeping the model efficient?

hard

A. 128 dimensions

B. 5000 dimensions

C. 10000 dimensions

D. 16 dimensions

Solution

1. Step 1: Consider vocabulary size and embedding size trade-off

   Very small embeddings (like 16) may miss details; very large (like 5000 or 10000) are costly and may overfit.

2. Step 2: Choose a moderate embedding size

   128 dimensions is a common practical choice balancing detail and efficiency for 10,000 words.

3. Final Answer:

   128 dimensions -> Option A

4. Quick Check:

   Moderate embedding size balances detail and efficiency [OK]

Hint: Pick moderate size like 128 for balance [OK]

Common Mistakes:

• Choosing too small embedding loses info
• Choosing too large wastes resources
• Matching embedding size to vocabulary size exactly