Embedding dimensionality considerations in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Embedding dimensionality considerations

Which metric matters for embedding dimensionality and WHY

When choosing embedding size, key metrics include model accuracy or task-specific performance (like classification accuracy or retrieval precision). This is because embedding size affects how well the model captures information. Too small, and the model misses details; too large, and it may overfit or slow down.

Also, training time and memory usage matter since bigger embeddings need more resources.

Confusion matrix or equivalent visualization

Embedding dimensionality itself does not produce a confusion matrix. Instead, we evaluate the downstream task using a confusion matrix. For example, if embeddings are used for classification, the confusion matrix shows true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN).

      Confusion Matrix Example:
      -------------------------
      |         | Pred Pos | Pred Neg |
      |---------|----------|----------|
      | True Pos|   TP=80  |   FN=20  |
      | True Neg|   FP=10  |   TN=90  |
      -------------------------
    

We compare confusion matrices for models with different embedding sizes to see which size yields better classification results.

Precision vs Recall tradeoff with concrete examples

Embedding size affects precision and recall indirectly by changing model quality.

Small embeddings: May miss important details, causing low recall (miss many true cases) but possibly high precision (few false alarms).

Large embeddings: Capture more detail, improving recall (find more true cases) but risk overfitting, which can lower precision (more false alarms).

Example: In a spam filter using embeddings, a small embedding might miss some spam emails (low recall), while a large embedding might flag many good emails as spam (low precision).

What "good" vs "bad" metric values look like for embedding dimensionality

Good: Balanced precision and recall with high overall accuracy or F1 score, reasonable training time, and manageable memory use.

Bad: Very low recall or precision, indicating embeddings are too small or too large; very long training times or memory errors due to too large embeddings.

For example, if a model with 50-dimensional embeddings has 85% accuracy and balanced precision/recall, but a 500-dimensional embedding model has 86% accuracy but takes 10x longer and uses much more memory, the smaller embedding might be better overall.

Metrics pitfalls

• Accuracy paradox: High accuracy with poor recall or precision can mislead about embedding quality.
• Overfitting: Very large embeddings may memorize training data, causing high training accuracy but poor test performance.
• Data leakage: If test data influences embedding training, metrics will be unrealistically high.
• Ignoring resource costs: Focusing only on accuracy without considering training time and memory can lead to impractical embedding sizes.

Self-check question

Your model uses 300-dimensional embeddings and achieves 98% accuracy but only 12% recall on the positive class (e.g., fraud). Is this good for production?

Answer: No. Despite high accuracy, the very low recall means the model misses most fraud cases. For fraud detection, recall is critical to catch as many frauds as possible. You should adjust embedding size or model to improve recall.

Key Result

Embedding size impacts model accuracy, recall, precision, and resource use; balance is key for good performance.

Practice

1. What does the dimensionality of an embedding vector mainly control in AI models?

easy

A. The color of the data points in visualization

B. The speed of the computer's processor

C. The level of detail or information captured about the item

D. The number of training examples needed

Solution

1. Step 1: Understand embedding vectors

   Embedding vectors represent items as numbers. Their length (dimensionality) decides how much detail they can hold.

2. Step 2: Relate dimensionality to information

   Higher dimensions mean more features can be captured, so more detail is stored about the item.

3. Final Answer:

   The level of detail or information captured about the item -> Option C

4. Quick Check:

   Embedding dimensionality = detail level [OK]

Hint: Embedding size = how detailed the vector is [OK]

Common Mistakes:

• Confusing dimensionality with training speed
• Thinking dimensionality affects data color
• Assuming dimensionality controls dataset size

2. Which of the following is the correct way to define an embedding layer with 50 dimensions in Python using PyTorch?

easy

A. nn.Embedding(dim=50, size=1000)

B. nn.Embedding(50, 1000)

C. nn.Embedding(embedding_size=50)

D. nn.Embedding(num_embeddings=1000, embedding_dim=50)

Solution

1. Step 1: Recall PyTorch embedding syntax

   PyTorch's embedding layer uses nn.Embedding(num_embeddings, embedding_dim).

2. Step 2: Match parameters to question

   We want 50 dimensions, so embedding_dim=50. Number of embeddings is usually vocabulary size, e.g., 1000.

3. Final Answer:

   nn.Embedding(num_embeddings=1000, embedding_dim=50) -> Option D

4. Quick Check:

   PyTorch embedding syntax = nn.Embedding(num_embeddings, embedding_dim) [OK]

Hint: Remember nn.Embedding(num_embeddings, embedding_dim) order [OK]

Common Mistakes:

• Swapping num_embeddings and embedding_dim
• Using wrong parameter names like dim or size
• Omitting required parameters

3. Consider this code snippet using TensorFlow to create embeddings:

embedding_layer = tf.keras.layers.Embedding(input_dim=5000, output_dim=16)
input_data = tf.constant([1, 2, 3])
output = embedding_layer(input_data)
print(output.shape)

What will be the printed shape?

medium

A. (3, 16)

B. (16, 3)

C. (3, 5000)

D. (5000, 16)

Solution

1. Step 1: Understand input and output dimensions

   Input is a list of 3 indices. Each index maps to a 16-dimensional vector.

2. Step 2: Determine output shape

   Output shape is (number of inputs, embedding dimension) = (3, 16).

3. Final Answer:

   (3, 16) -> Option A

4. Quick Check:

   Output shape = (input length, embedding dim) [OK]

Hint: Output shape = input count x embedding size [OK]

Common Mistakes:

• Confusing embedding dimension with input dimension
• Swapping rows and columns in output shape
• Assuming output shape equals input_dim

4. You have an embedding layer defined as nn.Embedding(1000, 128) in PyTorch. You try to pass an input tensor with values outside the range 0-999. What error will most likely occur?

medium

A. TypeError because input is not a float

B. IndexError due to out-of-range indices

C. ValueError because embedding dimension is wrong

D. No error, embeddings handle any input values

Solution

1. Step 1: Understand embedding input constraints

   Embedding layers expect input indices between 0 and num_embeddings-1 (0 to 999 here).

2. Step 2: Identify error from invalid indices

   Passing indices outside this range causes an IndexError because the layer cannot find embeddings for invalid indices.

3. Final Answer:

   IndexError due to out-of-range indices -> Option B

4. Quick Check:

   Embedding input indices must be valid [OK]

Hint: Embedding inputs must be valid indices [OK]

Common Mistakes:

• Thinking embeddings accept any numeric input
• Confusing input type errors with index errors
• Assuming embedding dimension affects input range

5. You want to choose the embedding dimensionality for a text classification model. The vocabulary size is 10,000 words. Which embedding size is the best balance between capturing enough detail and keeping the model efficient?

hard

A. 128 dimensions

B. 5000 dimensions

C. 10000 dimensions

D. 16 dimensions

Solution

1. Step 1: Consider vocabulary size and embedding size trade-off

   Very small embeddings (like 16) may miss details; very large (like 5000 or 10000) are costly and may overfit.

2. Step 2: Choose a moderate embedding size

   128 dimensions is a common practical choice balancing detail and efficiency for 10,000 words.

3. Final Answer:

   128 dimensions -> Option A

4. Quick Check:

   Moderate embedding size balances detail and efficiency [OK]

Hint: Pick moderate size like 128 for balance [OK]

Common Mistakes:

• Choosing too small embedding loses info
• Choosing too large wastes resources
• Matching embedding size to vocabulary size exactly