Context window handling in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding context window size impact

In natural language processing, what is the main effect of increasing the context window size when training a language model?

A The model can consider more words at once, improving understanding of long-range dependencies.

B The model trains faster because it processes fewer tokens per batch.

C The model reduces memory usage by limiting the number of tokens it stores.

D The model ignores earlier words and focuses only on the last few tokens.

💡 Hint

Think about how much text the model can see at once.

Increasing the context window size allows the model to see more words together, which helps it understand relationships between words that are far apart in the text.

❓ Predict Output

intermediate

Output of sliding window tokenization

What is the output of the following Python code that simulates a sliding context window over tokens?

NLP

tokens = ['I', 'love', 'machine', 'learning', 'and', 'AI']
window_size = 3
windows = [tokens[i:i+window_size] for i in range(len(tokens) - window_size + 1)]
print(windows)

A [['I', 'love', 'machine', 'learning'], ['love', 'machine', 'learning', 'and'], ['machine', 'learning', 'and', 'AI']]

B [['I', 'love', 'machine'], ['machine', 'learning', 'and'], ['love', 'machine', 'learning'], ['learning', 'and', 'AI']]

C [['I', 'love'], ['love', 'machine'], ['machine', 'learning'], ['learning', 'and'], ['and', 'AI']]

D [['I', 'love', 'machine'], ['love', 'machine', 'learning'], ['machine', 'learning', 'and'], ['learning', 'and', 'AI']]

💡 Hint

Look at how the list comprehension slices the tokens with a fixed window size.

The code creates overlapping windows of size 3 moving one token at a time, so each window contains 3 consecutive tokens.

❓ Model Choice

advanced

Choosing model type for long context handling

You want to build a model that can understand very long documents (thousands of words). Which model architecture is best suited to handle such long context windows efficiently?

A Transformer with sparse attention or memory-augmented mechanisms

B Recurrent Neural Network (RNN) with truncated backpropagation

C Standard Transformer with fixed 512-token context window

D Feedforward neural network without sequence modeling

💡 Hint

Consider models designed to reduce computation for long sequences.

Transformers with sparse attention or memory mechanisms can handle longer contexts by focusing attention on important parts, reducing computation compared to standard transformers.

❓ Hyperparameter

advanced

Effect of context window size on training time

How does increasing the context window size affect the training time of a Transformer-based language model?

A Training time decreases because the model sees more tokens at once.

B Training time increases roughly quadratically with context window size.

C Training time remains constant regardless of context window size.

D Training time increases linearly with context window size.

💡 Hint

Think about how attention computation scales with sequence length.

Self-attention in transformers computes relationships between all token pairs, so computation grows roughly with the square of the context window size.

🔧 Debug

expert

Identifying error in context window implementation

What error does the following code raise when trying to create context windows, and why?

tokens = ['a', 'b', 'c']
window_size = 4
windows = [tokens[i:i+window_size] for i in range(len(tokens) - window_size + 1)]
print(windows)

A IndexError because slicing goes beyond the list length.

B ValueError because window_size must be less than tokens length.

C Empty list because the range is negative, so no windows are created.

D TypeError because window_size is larger than tokens length.

💡 Hint

Check what happens when the range argument is negative.

The range becomes range(3 - 4 + 1) = range(0), which is empty, so windows is an empty list without error.

Practice

1. What does the term context window mean in natural language processing?

easy

A. A method to remove stop words from text

B. The entire document used for training a model

C. A list of all words in a sentence

D. A small part of text around a word used to understand its meaning

Solution

1. Step 1: Understand the definition of context window

   The context window refers to a limited number of words surrounding a target word to help understand its meaning.

2. Step 2: Compare options with the definition

   Only A small part of text around a word used to understand its meaning correctly describes this as a small part of text around a word. Other options describe unrelated concepts.

3. Final Answer:

   A small part of text around a word used to understand its meaning -> Option D

4. Quick Check:

   Context window = small text part around word [OK]

Hint: Context window = nearby words around a target word [OK]

Common Mistakes:

• Confusing context window with entire document
• Thinking it means all words in a sentence
• Mixing it up with stop word removal

2. Which of the following is the correct way to define a context window of size 3 around the word at index 5 in a list words?

easy

A. words[4:7]

B. words[3:8]

C. words[2:7]

D. words[5:8]

Solution

1. Step 1: Understand context window size and indexing

   A window size of 3 means 3 words total, usually centered on the target word. For index 5, the window covers indices 4, 5, 6.

2. Step 2: Check each option's slice range

   words[4:7] slices words[4:7], which includes indices 4, 5, 6 (3 words). Others include wrong ranges or counts.

3. Final Answer:

   words[4:7] -> Option A

4. Quick Check:

   Window size 3 around index 5 = indices 4 to 6 [OK]

Hint: Slice from index-1 to index+2 for window size 3 [OK]

Common Mistakes:

• Using wrong slice indices causing off-by-one errors
• Including too many or too few words
• Not centering window on target word

3. Given the code below, what will be the output?

words = ['I', 'love', 'to', 'eat', 'apples', 'and', 'bananas']
index = 4
window_size = 3
start = max(0, index - window_size // 2)
end = min(len(words), index + window_size // 2 + 1)
context = words[start:end]
print(context)

medium

A. ['to', 'eat', 'apples']

B. ['eat', 'apples', 'and']

C. ['apples', 'and', 'bananas']

D. ['love', 'to', 'eat']

Solution

1. Step 1: Calculate start and end indices

   window_size is 3, so window_size // 2 = 1. start = max(0, 4 - 1) = 3, end = min(7, 4 + 1 + 1) = 6.

2. Step 2: Extract words from start to end

   words[3:6] = ['eat', 'apples', 'and'].

3. Final Answer:

   ['eat', 'apples', 'and'] -> Option B

4. Quick Check:

   Slice words[3:6] = ['eat', 'apples', 'and'] [OK]

Hint: Calculate start/end with floor division and slice accordingly [OK]

Common Mistakes:

• Off-by-one errors in slicing
• Ignoring max/min boundaries
• Misunderstanding integer division

4. The following code tries to get a context window but sometimes throws an error. What is the main issue?

def get_context(words, index, window_size):
    start = index - window_size // 2
    end = index + window_size // 2 + 1
    return words[start:end]

words = ['hello', 'world']
print(get_context(words, 0, 3))

medium

A. index is out of range

B. window_size must be even

C. start can be negative causing an IndexError

D. The function does not return a list

Solution

1. Step 1: Analyze start index calculation

   For index=0 and window_size=3, start = 0 - 1 = -1, which is negative.

2. Step 2: Understand Python slicing with negative start

   Negative start in slicing accesses from the end, which may cause unexpected results or errors if out of range.

3. Final Answer:

   start can be negative causing an IndexError -> Option C

4. Quick Check:

   Negative start index causes slicing issues [OK]

Hint: Check if start index is negative before slicing [OK]

Common Mistakes:

• Assuming negative indices always work safely
• Thinking window_size must be even
• Ignoring index bounds

5. You want to build a model that uses a context window of size 5 to understand words in sentences. Which approach best handles sentences shorter than 5 words without errors?

hard

A. Pad the sentence with special tokens to length 5 before extracting the window

B. Always extract 5 words ignoring sentence length, causing errors if too short

C. Use only the first word as context if sentence is short

D. Skip sentences shorter than 5 words during training

Solution

1. Step 1: Understand the problem with short sentences

   Sentences shorter than the window size cause indexing errors or incomplete context.

2. Step 2: Evaluate options for handling short sentences

   Padding with special tokens ensures fixed length and avoids errors, unlike skipping or ignoring length.

3. Final Answer:

   Pad the sentence with special tokens to length 5 before extracting the window -> Option A

4. Quick Check:

   Padding fixes short sentence context window issues [OK]

Hint: Pad short sentences to window size to avoid errors [OK]

Common Mistakes:

• Ignoring short sentences causing runtime errors
• Skipping data reduces training quality
• Using incomplete context weakens model understanding