Embedding dimensionality considerations in Prompt Engineering / GenAI - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create an embedding layer with the correct input dimension.

Prompt Engineering / GenAI

embedding_layer = Embedding(input_dim=[1], output_dim=50)

Drag options to blanks, or click blank then click option'

A1000

B50

C10

D500

Attempts:

3 left

💡 Hint

Common Mistakes

Using output_dim instead of input_dim for vocabulary size.

Confusing embedding size with vocabulary size.

The input_dim should be the size of the vocabulary, which is 1000 in this example.

2fill in blank

medium

Complete the code to set the embedding output dimension to 128.

Prompt Engineering / GenAI

embedding_layer = Embedding(input_dim=1000, output_dim=[1])

Drag options to blanks, or click blank then click option'

A256

B64

C128

D32

Attempts:

3 left

💡 Hint

Common Mistakes

Confusing input_dim with output_dim.

Choosing too small or too large embedding size without reason.

The output_dim is the size of each embedding vector, set here to 128.

3fill in blank

hard

Fix the error in the code to correctly initialize an embedding layer with vocabulary size 5000 and embedding size 100.

Prompt Engineering / GenAI

embedding_layer = Embedding([1], output_dim=100)

Drag options to blanks, or click blank then click option'

Ainput_shape=5000

Boutput_dim=5000

Cinput_length=5000

Dinput_dim=5000

Attempts:

3 left

💡 Hint

Common Mistakes

Using output_dim instead of input_dim for vocabulary size.

Using input_length or input_shape incorrectly.

The correct parameter for vocabulary size is input_dim=5000.

4fill in blank

hard

Fill both blanks to create an embedding layer with vocabulary size 2000 and embedding size 64.

Prompt Engineering / GenAI

embedding_layer = Embedding([1], output_dim=[2])

Drag options to blanks, or click blank then click option'

Ainput_dim=2000

Binput_dim=64

C64

D2000

Attempts:

3 left

💡 Hint

Common Mistakes

Swapping input_dim and output_dim values.

Using numeric values without parameter names for input_dim.

input_dim=2000 sets vocabulary size; output_dim=64 sets embedding size.

5fill in blank

hard

Fill all three blanks to create an embedding layer with vocabulary size 3000, embedding size 128, and input length 50.

Prompt Engineering / GenAI

embedding_layer = Embedding([1], [2], input_length=[3])

Drag options to blanks, or click blank then click option'

Ainput_dim=3000

Boutput_dim=128

C50

Dinput_length=50

Attempts:

3 left

💡 Hint

Common Mistakes

Using input_length without parameter name.

Confusing output_dim with input_dim.

Putting input_length in wrong position.

input_dim=3000 sets vocabulary size, output_dim=128 sets embedding size, input_length=50 sets input sequence length.

Practice

1. What does the dimensionality of an embedding vector mainly control in AI models?

easy

A. The color of the data points in visualization

B. The speed of the computer's processor

C. The level of detail or information captured about the item

D. The number of training examples needed

Solution

1. Step 1: Understand embedding vectors

   Embedding vectors represent items as numbers. Their length (dimensionality) decides how much detail they can hold.

2. Step 2: Relate dimensionality to information

   Higher dimensions mean more features can be captured, so more detail is stored about the item.

3. Final Answer:

   The level of detail or information captured about the item -> Option C

4. Quick Check:

   Embedding dimensionality = detail level [OK]

Hint: Embedding size = how detailed the vector is [OK]

Common Mistakes:

• Confusing dimensionality with training speed
• Thinking dimensionality affects data color
• Assuming dimensionality controls dataset size

2. Which of the following is the correct way to define an embedding layer with 50 dimensions in Python using PyTorch?

easy

A. nn.Embedding(dim=50, size=1000)

B. nn.Embedding(50, 1000)

C. nn.Embedding(embedding_size=50)

D. nn.Embedding(num_embeddings=1000, embedding_dim=50)

Solution

1. Step 1: Recall PyTorch embedding syntax

   PyTorch's embedding layer uses nn.Embedding(num_embeddings, embedding_dim).

2. Step 2: Match parameters to question

   We want 50 dimensions, so embedding_dim=50. Number of embeddings is usually vocabulary size, e.g., 1000.

3. Final Answer:

   nn.Embedding(num_embeddings=1000, embedding_dim=50) -> Option D

4. Quick Check:

   PyTorch embedding syntax = nn.Embedding(num_embeddings, embedding_dim) [OK]

Hint: Remember nn.Embedding(num_embeddings, embedding_dim) order [OK]

Common Mistakes:

• Swapping num_embeddings and embedding_dim
• Using wrong parameter names like dim or size
• Omitting required parameters

3. Consider this code snippet using TensorFlow to create embeddings:

embedding_layer = tf.keras.layers.Embedding(input_dim=5000, output_dim=16)
input_data = tf.constant([1, 2, 3])
output = embedding_layer(input_data)
print(output.shape)

What will be the printed shape?

medium

A. (3, 16)

B. (16, 3)

C. (3, 5000)

D. (5000, 16)

Solution

1. Step 1: Understand input and output dimensions

   Input is a list of 3 indices. Each index maps to a 16-dimensional vector.

2. Step 2: Determine output shape

   Output shape is (number of inputs, embedding dimension) = (3, 16).

3. Final Answer:

   (3, 16) -> Option A

4. Quick Check:

   Output shape = (input length, embedding dim) [OK]

Hint: Output shape = input count x embedding size [OK]

Common Mistakes:

• Confusing embedding dimension with input dimension
• Swapping rows and columns in output shape
• Assuming output shape equals input_dim

4. You have an embedding layer defined as nn.Embedding(1000, 128) in PyTorch. You try to pass an input tensor with values outside the range 0-999. What error will most likely occur?

medium

A. TypeError because input is not a float

B. IndexError due to out-of-range indices

C. ValueError because embedding dimension is wrong

D. No error, embeddings handle any input values

Solution

1. Step 1: Understand embedding input constraints

   Embedding layers expect input indices between 0 and num_embeddings-1 (0 to 999 here).

2. Step 2: Identify error from invalid indices

   Passing indices outside this range causes an IndexError because the layer cannot find embeddings for invalid indices.

3. Final Answer:

   IndexError due to out-of-range indices -> Option B

4. Quick Check:

   Embedding input indices must be valid [OK]

Hint: Embedding inputs must be valid indices [OK]

Common Mistakes:

• Thinking embeddings accept any numeric input
• Confusing input type errors with index errors
• Assuming embedding dimension affects input range

5. You want to choose the embedding dimensionality for a text classification model. The vocabulary size is 10,000 words. Which embedding size is the best balance between capturing enough detail and keeping the model efficient?

hard

A. 128 dimensions

B. 5000 dimensions

C. 10000 dimensions

D. 16 dimensions

Solution

1. Step 1: Consider vocabulary size and embedding size trade-off

   Very small embeddings (like 16) may miss details; very large (like 5000 or 10000) are costly and may overfit.

2. Step 2: Choose a moderate embedding size

   128 dimensions is a common practical choice balancing detail and efficiency for 10,000 words.

3. Final Answer:

   128 dimensions -> Option A

4. Quick Check:

   Moderate embedding size balances detail and efficiency [OK]

Hint: Pick moderate size like 128 for balance [OK]

Common Mistakes:

• Choosing too small embedding loses info
• Choosing too large wastes resources
• Matching embedding size to vocabulary size exactly