Why sequence models understand word order in NLP - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why do sequence models like RNNs process words in order?

Sequence models such as RNNs read words one by one in a sentence. Why is this order important?

A Because the model ignores previous words and treats each word independently.

B Because the model uses previous words' information to understand the current word's meaning.

C Because the model randomly shuffles words before processing to increase variety.

D Because the model processes all words simultaneously without any order.

💡 Hint

Think about how understanding a sentence depends on the words that came before.

RNNs keep a memory of previous words, so the order matters to capture context and meaning.

❓ Predict Output

intermediate

Output of a simple RNN processing word indices

Given a simple RNN that processes a sequence of word indices, what is the output shape after processing a batch of 2 sequences each of length 3?

NLP

import torch
import torch.nn as nn

rnn = nn.RNN(input_size=5, hidden_size=4, batch_first=True)
inputs = torch.randn(2, 3, 5)  # batch=2, seq_len=3, input_size=5
output, hn = rnn(inputs)
print(output.shape)

A torch.Size([2, 3, 4])

B torch.Size([3, 2, 4])

C torch.Size([2, 4, 3])

D torch.Size([4, 3, 2])

💡 Hint

Remember batch_first=True means batch is the first dimension.

The output shape is (batch_size, sequence_length, hidden_size) = (2, 3, 4).

❓ Model Choice

advanced

Which model inherently captures word order in text?

Among these models, which one inherently understands the order of words in a sentence without additional position information?

A Transformer without positional encoding

B Bag-of-Words model

C Multilayer Perceptron (MLP) on word counts

D Recurrent Neural Network (RNN)

💡 Hint

Think about which model processes words one after another.

RNNs process words sequentially, so they naturally capture word order. Transformers need positional encoding, and Bag-of-Words or MLP on counts ignore order.

❓ Hyperparameter

advanced

Effect of sequence length on RNN training

What is a common problem when training RNNs on very long sequences, and which hyperparameter adjustment can help?

A Vanishing gradients; use gradient clipping to limit gradient size.

B Underfitting; reduce hidden size to simplify the model.

C Overfitting; increase learning rate to speed training.

D Data leakage; shuffle sequences randomly before training.

💡 Hint

Think about what happens to gradients when backpropagating through many steps.

Long sequences cause gradients to vanish or explode. Gradient clipping helps by limiting gradient size to stabilize training.

❓ Metrics

expert

Interpreting perplexity in language models

A language model has a perplexity of 50 on a test set. What does this number mean?

A The model predicts the next word correctly 50 times in a row.

B The model makes 50% errors predicting the next word.

C On average, the model is as uncertain as choosing among 50 equally likely next words.

D The model's loss value is 50 after training.

💡 Hint

Perplexity measures how well a model predicts a sequence; lower is better.

Perplexity of 50 means the model is as confused as if it had to pick from 50 equally likely options for the next word.