Why sequence models understand word order in NLP - Test Your Understanding

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a sequence model layer that processes input words in order.

NLP

from tensorflow.keras.layers import [1]
sequence_layer = [1](units=32)

Drag options to blanks, or click blank then click option'

AConv2D

BLSTM

CDense

DFlatten

Attempts:

3 left

💡 Hint

Common Mistakes

Choosing Dense or Conv2D which do not handle sequence order.

The LSTM layer is a type of sequence model that processes data in order, capturing word order information.

2fill in blank

medium

Complete the code to add positional encoding to input embeddings to help the model understand word order.

NLP

import tensorflow as tf
import numpy as np

def positional_encoding(seq_len, d_model):
    pos = np.arange(seq_len)[:, np.newaxis]
    i = np.arange(d_model)[np.newaxis, :]
    angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))
    angle_rads = pos * angle_rates
    sines = np.sin(angle_rads[:, 0::2])
    cosines = np.cos(angle_rads[:, 1::2])
    pos_encoding = np.concatenate([sines, cosines], axis=-1)
    return tf.cast(pos_encoding, dtype=tf.float32)

seq_len = 50
d_model = 128
pos_encoding = positional_encoding(seq_len, d_model)

input_embeddings = tf.random.uniform((1, seq_len, d_model))
output = input_embeddings + [1]

Drag options to blanks, or click blank then click option'

Atf.random.normal(input_embeddings.shape)

Btf.zeros_like(input_embeddings)

Cpos_encoding

Dtf.ones_like(input_embeddings)

Attempts:

3 left

💡 Hint

Common Mistakes

Adding zeros or random noise instead of positional encoding.

Adding positional encoding to embeddings helps the model know the position of each word in the sequence.

3fill in blank

hard

Fix the error in the code that tries to create a Transformer model input layer for sequences.

NLP

from tensorflow.keras.layers import Input

sequence_input = Input(shape=([1],), dtype='int32')

Drag options to blanks, or click blank then click option'

A100

B1

C32

DNone

Attempts:

3 left

💡 Hint

Common Mistakes

Using fixed sequence length which restricts input flexibility.

The input shape for sequences should have None as the sequence length to allow variable length inputs.

4fill in blank

hard

Fill both blanks to create a dictionary comprehension that maps words to their lengths only if length is greater than 3.

NLP

words = ['apple', 'cat', 'banana', 'dog']
lengths = {word: [1] for word in words if [2]

Drag options to blanks, or click blank then click option'

Alen(word)

Blen(word) > 3

Cword.startswith('a')

Dword == 'banana'

Attempts:

3 left

💡 Hint

Common Mistakes

Using wrong condition or value expression.

This comprehension creates a dictionary with words as keys and their lengths as values, but only for words longer than 3 letters.

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps uppercase words to their lengths if length is less than 6.

NLP

words = ['apple', 'cat', 'banana', 'dog']
result = { [1]: [2] for word in words if [3] }

Drag options to blanks, or click blank then click option'

Aword.upper()

Blen(word)

Clen(word) < 6

Dword.lower()

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing up keys and values or wrong condition.

This comprehension creates a dictionary with uppercase words as keys and their lengths as values, only for words shorter than 6 letters.

Practice

1. Why do sequence models like LSTM and GRU understand word order in sentences?

easy

A. Because they only look at the first word in a sentence

B. Because they treat all words independently without order

C. Because they process words one after another, keeping track of order

D. Because they randomly shuffle words before processing

Solution

1. Step 1: Understand sequence model processing

   Sequence models process input data step-by-step, maintaining information about previous words.

2. Step 2: Recognize how order is preserved

   This stepwise processing allows the model to remember the order of words, which is crucial for meaning.

3. Final Answer:

   Because they process words one after another, keeping track of order -> Option C

4. Quick Check:

   Sequence models = process words in order [OK]

Hint: Sequence models read words stepwise to keep order [OK]

Common Mistakes:

• Thinking models treat words independently
• Assuming models ignore word order
• Believing models shuffle words randomly

2. Which of the following is the correct way to describe how an LSTM processes a sentence?

easy

A. It processes words sequentially, updating its memory at each step

B. It randomly selects words to process in any order

C. It ignores previous words and only looks at the current word

D. It processes all words simultaneously without order

Solution

1. Step 1: Recall LSTM processing method

   LSTM processes input words one by one, updating its internal state to remember past information.

2. Step 2: Confirm sequential update of memory

   This sequential update allows LSTM to capture word order and context effectively.

3. Final Answer:

   It processes words sequentially, updating its memory at each step -> Option A

4. Quick Check:

   LSTM = sequential processing with memory update [OK]

Hint: LSTM updates memory step-by-step in word order [OK]

Common Mistakes:

• Thinking LSTM processes all words at once
• Believing LSTM ignores previous words
• Assuming random word processing

3. Consider this simplified code snippet of a sequence model processing words:

words = ['I', 'love', 'AI']
state = 0
for word in words:
    state += len(word)
print(state)

What will be the output?

medium

A. 6

B. 9

C. 8

D. 7

Solution

1. Step 1: Calculate length of each word

   'I' has length 1, 'love' has length 4, 'AI' has length 2.

2. Step 2: Sum lengths in the loop

   state = 0 + 1 + 4 + 2 = 7; 1 + 4 = 5, 5 + 2 = 7.

3. Step 3: Verify code logic

   Code adds len(word) to state for each word: 'I'(1), 'love'(4), 'AI'(2). Sum is 7, so output is 7.

4. Final Answer:

   7 -> Option D

5. Quick Check:

   Sum of word lengths = 7 [OK]

Hint: Add lengths of each word in order [OK]

Common Mistakes:

• Adding number of words instead of lengths
• Miscounting word lengths
• Ignoring the loop accumulation

4. This code tries to simulate a sequence model but has a bug:

words = ['hello', 'world']
state = 0
for i in range(len(words)):
    state = len(words[i])  # Bug here
print(state)

What is the bug and how to fix it?

medium

A. Bug: state is overwritten each time; Fix: use state += len(words[i])

B. Bug: range should be range(words); Fix: change loop to for word in words

C. Bug: len(words[i]) is wrong; Fix: use len(words)

D. Bug: print(state) is outside loop; Fix: move print inside loop

Solution

1. Step 1: Identify the bug in state update

   The code sets state = len(words[i]) each loop, overwriting previous value instead of accumulating.

2. Step 2: Fix by accumulating lengths

   Change to state += len(words[i]) to add lengths instead of replacing state.

3. Final Answer:

   Bug: state is overwritten each time; Fix: use state += len(words[i]) -> Option A

4. Quick Check:

   Use += to accumulate state [OK]

Hint: Use += to add, not = to overwrite [OK]

Common Mistakes:

• Overwriting state instead of adding
• Changing loop incorrectly
• Moving print unnecessarily

5. You want to build a model that understands the sentence meaning by considering word order. Which approach best captures this?

hard

A. Use a bag-of-words model that counts word frequency ignoring order

B. Use a sequence model like LSTM that processes words in order

C. Use a model that randomly shuffles words before processing

D. Use a model that only looks at the last word in the sentence

Solution

1. Step 1: Understand model types and word order

   Bag-of-words ignores order; sequence models like LSTM process words in order.

2. Step 2: Choose model that captures order for meaning

   LSTM captures word order and context, making it best for sentence meaning.

3. Final Answer:

   Use a sequence model like LSTM that processes words in order -> Option B

4. Quick Check:

   Sequence model = best for word order [OK]

Hint: Choose sequence models to keep word order [OK]

Common Mistakes:

• Choosing bag-of-words which ignores order
• Thinking random shuffle helps
• Using only last word loses context