NLP - Sequence Models for NLP
Given query Q = [1, 2], key K = [[1, 0], [0, 1]], and value V = [[10, 0], [0, 20]], what is the attention output using dot product and softmax?
Given query Q = [1, 2], key K = [[1, 0], [0, 1]], and value V = [[10, 0], [0, 20]], what is the attention output using dot product and softmax?
medium📝 Predict Output Q4 of 15
Step-by-Step Solution
Solution:
Step 1: Compute dot products of Q with each key
Dot(Q, K1) = 1*1 + 2*0 = 1; Dot(Q, K2) = 1*0 + 2*1 = 2.Step 2: Apply softmax to scores [1, 2]
Softmax(1,2) = [e^1/(e^1+e^2), e^2/(e^1+e^2)] ≈ [0.269, 0.731].Step 3: Calculate weighted sum of values
Output ≈ 0.269*[10,0] + 0.731*[0,20] = [2.69, 14.62].Final Answer:
[2.69, 14.62] -> Option AQuick Check:
Attention output ≈ weighted sum of values [OK]
Quick Trick: Softmax weights times values gives output [OK]
Common Mistakes:
MISTAKES- Forgetting softmax
- Mixing key and value
- Wrong dot product order
Master "Sequence Models for NLP" in NLP
Want More Practice?
15+ quiz questions · All difficulty levels · Free
Free Signup - Practice All QuestionsMore NLP Quizzes