Transformer architecture in NLP - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a Transformer encoder layer using PyTorch.

NLP

import torch.nn as nn
encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=[1])

Drag options to blanks, or click blank then click option'

A16

B4

C8

D32

Attempts:

3 left

💡 Hint

Common Mistakes

Choosing a number of heads that does not divide the model dimension evenly.

The number of attention heads (nhead) is commonly set to 8 in Transformer models for balanced performance.

2fill in blank

medium

Complete the code to apply positional encoding to the input embeddings.

NLP

pos_encoded = embeddings + [1]

Drag options to blanks, or click blank then click option'

Ann.Embedding(num_positions, embedding_dim)

Bpositional_encoding

Cnn.Linear(embedding_dim, embedding_dim)

Dnn.LayerNorm(embedding_dim)

Attempts:

3 left

💡 Hint

Common Mistakes

Using a layer like nn.Embedding or nn.Linear instead of a positional encoding tensor.

Positional encoding is added to embeddings to give the model information about token positions.

3fill in blank

hard

Fix the error in the multi-head attention call by filling the correct argument.

NLP

output, weights = multihead_attn(query, key, [1])

Drag options to blanks, or click blank then click option'

Avalue

Bmask

Ckey_padding_mask

Dattn_mask

Attempts:

3 left

💡 Hint

Common Mistakes

Passing mask or key_padding_mask as the third positional argument instead of value.

The third argument to multihead_attn is the value tensor, which is used along with query and key.

4fill in blank

hard

Fill both blanks to complete the Transformer decoder layer initialization.

NLP

decoder_layer = nn.TransformerDecoderLayer(d_model=[1], nhead=[2])

Drag options to blanks, or click blank then click option'

A512

B256

C8

D4

Attempts:

3 left

💡 Hint

Common Mistakes

Mismatching d_model and nhead values that don't align with embedding size.

The decoder layer uses d_model=512 and nhead=8 to match common Transformer settings.

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps each token to its embedding size if the size is greater than 300.

NLP

embedding_sizes = {token: [1] for token, size in token_sizes.items() if size [2] 300 and size == [3]

Drag options to blanks, or click blank then click option'

Asize

B>

C512

Dtoken

Attempts:

3 left

💡 Hint

Common Mistakes

Using token instead of size as dictionary value or wrong comparison operators.

This comprehension selects tokens with embedding size greater than 300 and exactly 512, mapping token to size.