Attention mechanism basics in NLP - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to compute the attention scores using dot product.

NLP

attention_scores = query [1] key.transpose(-2, -1)

Drag options to blanks, or click blank then click option'

A@

B+

C*

D-

Attempts:

3 left

💡 Hint

Common Mistakes

Using '*' which does element-wise multiplication, not matrix multiplication.

Using '+' or '-' which are arithmetic operators not for multiplication.

The attention scores are computed by the dot product of the query and the key vectors. In Python, the '@' operator performs matrix multiplication.

2fill in blank

medium

Complete the code to apply softmax to the attention scores along the last dimension.

NLP

attention_weights = torch.nn.functional.softmax(attention_scores, dim=[1])

Drag options to blanks, or click blank then click option'

A0

B1

C2

D-1

Attempts:

3 left

💡 Hint

Common Mistakes

Applying softmax along the wrong dimension, which changes the meaning of weights.

Using 0 or 1 which may not correspond to the keys dimension.

Softmax is applied along the last dimension (-1) to convert scores into probabilities for each key.

3fill in blank

hard

Fix the error in scaling the attention scores by the square root of the key dimension.

NLP

scaled_scores = attention_scores / torch.sqrt(torch.tensor([1], dtype=torch.float32))

Drag options to blanks, or click blank then click option'

Akey.shape[0]

Bkey.size(-1)

Cquery.size(0)

Dquery.shape[1]

Attempts:

3 left

💡 Hint

Common Mistakes

Using batch size or query dimensions instead of key vector dimension.

Using shape[0] which is usually batch size, not feature size.

The scaling factor is the square root of the key vector dimension, which is the last dimension size of the key tensor.

4fill in blank

hard

Fill both blanks to compute the attention output by multiplying attention weights with values.

NLP

attention_output = torch.matmul([1], [2])

Drag options to blanks, or click blank then click option'

Aattention_weights

Bvalues

Cquery

Dkey

Attempts:

3 left

💡 Hint

Common Mistakes

Multiplying query or key instead of values.

Swapping the order of multiplication.

The attention output is the weighted sum of the value vectors, so we multiply attention weights by values.

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps each word to its length if length is greater than 3.

NLP

lengths = { [1] : [2] for [3] in words if len([3]) > 3 }

Drag options to blanks, or click blank then click option'

Aword

Blen(word)

Ditem

Attempts:

3 left

💡 Hint

Common Mistakes

Using the wrong variable name in the loop.

Using the word as value instead of its length.

This dictionary comprehension creates a mapping from each word to its length, but only for words longer than 3 characters.