Embedding dimensionality considerations in Prompt Engineering / GenAI - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is embedding dimensionality in machine learning?

Embedding dimensionality is the number of numbers used to represent each item (like a word or image) in a vector form. It controls how much detail the model can capture about that item.

intermediate

Why does increasing embedding dimensionality not always improve model performance?

Higher dimensionality can capture more details but may cause overfitting, where the model learns noise instead of useful patterns. It also needs more data and computing power.

beginner

How does embedding dimensionality affect computational cost?

Larger embedding dimensions mean bigger vectors, which require more memory and slower calculations during training and prediction.

intermediate

What is a common rule of thumb for choosing embedding dimensionality?

A simple rule is to start with the fourth root of the vocabulary size for word embeddings, then adjust based on model performance and resources.

beginner

What happens if embedding dimensionality is too low?

If too low, embeddings may not capture enough information, leading to poor model understanding and lower accuracy.

What does embedding dimensionality represent?

A The number of features in each embedding vector

B The number of training samples

C The number of output classes

D The learning rate of the model

Embedding dimensionality is the size of the vector used to represent each item.

What is a risk of using very high embedding dimensionality?

A Underfitting the data

B Overfitting and increased computation

C Faster training

D Reduced model size

High dimensionality can cause overfitting and requires more computing resources.

Which of these is a sign that embedding dimensionality might be too low?

A Model overfits quickly

B Training is very slow

C Model accuracy is very low

D Embeddings use too much memory

Low dimensionality can limit the model's ability to learn, causing low accuracy.

How does embedding dimensionality affect memory usage?

A Higher dimensionality uses more memory

B Higher dimensionality uses less memory

C It has no effect on memory

D Memory usage depends only on batch size

Larger embeddings require more memory to store.

What is a simple starting point to choose embedding size for words?

A Square root of vocabulary size

B Vocabulary size divided by 10

C Fixed size of 100 always

D Fourth root of vocabulary size

A common rule is to use the fourth root of the vocabulary size as a starting embedding dimension.

Explain why embedding dimensionality is important and how it affects model performance and resource use.

Think about balancing detail and resources.

Describe a practical approach to selecting embedding dimensionality for a new dataset.

Start simple, then tune.