Bird
0
0

Why is the dot product in scaled dot-product attention divided by the square root of the key dimension (d_k) before applying softmax?

hard📝 Application Q9 of 15
NLP - Sequence Models for NLP
Why is the dot product in scaled dot-product attention divided by the square root of the key dimension (d_k) before applying softmax?
ATo prevent the dot product values from becoming too large, which can cause gradients to vanish during training.
BTo increase the magnitude of dot products, making softmax outputs more confident.
CTo normalize the key vectors to unit length before computing attention.
DTo reduce the computational complexity of the attention mechanism.
Step-by-Step Solution
Solution:

  1. Step 1: Understand dot product scale

    Dot products grow with vector dimension, causing large values.

  2. Step 2: Effect on softmax

    Large values cause softmax to saturate, leading to very small gradients.

  3. Step 3: Scaling purpose

    Dividing by sqrt(d_k) keeps values in a range that stabilizes gradients.

  4. Final Answer:

    To prevent the dot product values from becoming too large, which can cause gradients to vanish during training. -> Option A

  5. Quick Check:

    Scaling prevents large values and gradient issues [OK]
Quick Trick: Scale dot product to avoid softmax saturation [OK]
Common Mistakes:
MISTAKES
  • Thinking scaling increases confidence
  • Confusing scaling with normalization of vectors
  • Assuming scaling reduces computation

Want More Practice?

15+ quiz questions · All difficulty levels · Free

Free Signup - Practice All Questions
More NLP Quizzes
Sentiment Analysis Advanced - Aspect-based sentiment analysis - Quiz 5medium Sequence Models for NLP - Embedding layer usage - Quiz 13medium Sequence Models for NLP - GRU for text - Quiz 2easy Sequence Models for NLP - Padding and sequence length - Quiz 7medium Text Similarity and Search - Semantic similarity with embeddings - Quiz 8hard Text Similarity and Search - Semantic similarity with embeddings - Quiz 11easy Topic Modeling - Choosing number of topics - Quiz 10hard Topic Modeling - LDA with scikit-learn - Quiz 8hard Word Embeddings - Pre-trained embedding usage - Quiz 11easy Word Embeddings - GloVe embeddings - Quiz 13medium