NLP - Sequence Models for NLP
Given query vector Q = [1, 0], key vectors K1 = [1, 0], K2 = [0, 1], and value vectors V1 = [10, 0], V2 = [0, 20], what is the attention output after applying softmax on Q·K^T and multiplying by values?
Given query vector Q = [1, 0], key vectors K1 = [1, 0], K2 = [0, 1], and value vectors V1 = [10, 0], V2 = [0, 20], what is the attention output after applying softmax on Q·K^T and multiplying by values?
medium📝 Predict Output Q13 of 15
Step-by-Step Solution
Solution:
Step 1: Calculate dot products Q·K1 and Q·K2
Q·K1 = 1*1 + 0*0 = 1; Q·K2 = 1*0 + 0*1 = 0.Step 2: Apply softmax to [1, 0]
Softmax(1,0) = [e^1/(e^1+e^0), e^0/(e^1+e^0)] ≈ [0.731, 0.269].Step 3: Multiply weights by values and sum
Output = 0.731*[10,0] + 0.269*[0,20] = [7.31, 0] + [0,5.38] = [7.31, 5.38].Step 4: Match to options
The computed output [7.31, 5.38] matches [7.31, 5.38] (approximate values).Final Answer:
[7.31, 5.38] -> Option CQuick Check:
Softmax weights x values = output [OK]
Quick Trick: Softmax weights times values gives attention output [OK]
Common Mistakes:
MISTAKES- Skipping softmax normalization
- Multiplying query with values directly
- Ignoring vector multiplication order
Master "Sequence Models for NLP" in NLP
Want More Practice?
15+ quiz questions · All difficulty levels · Free
Free Signup - Practice All QuestionsMore NLP Quizzes