Context window handling in NLP - ML Experiment: Train & Evaluate

Experiment - Context window handling

Problem:You are training a simple text classification model using a fixed-size context window of 5 words. The model currently uses only the immediate 5 words before the target word to predict its class.

Current Metrics:Training accuracy: 92%, Validation accuracy: 75%

Issue:The model shows overfitting and poor generalization because the context window is too small to capture enough information, leading to low validation accuracy.

Your Task

Increase the context window size to improve validation accuracy to at least 85% while keeping training accuracy below 95% to reduce overfitting.

You can only change the context window size and retrain the model.

Do not change the model architecture or dataset.

Hint 1

Hint 2

Hint 3

Solution

NLP

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Sample data
texts = [
    'the cat sat on the mat',
    'dogs are great pets',
    'the quick brown fox jumps',
    'lorem ipsum dolor sit amet',
    'machine learning is fun',
    'natural language processing',
    'deep learning models',
    'artificial intelligence',
    'data science and ai',
    'python programming language'
]
labels = [0, 1, 0, 1, 0, 1, 0, 1, 0, 1]  # Binary labels

# Function to create context windows
def create_context_windows(texts, window_size):
    new_texts = []
    for text in texts:
        words = text.split()
        for i in range(len(words)):
            start = max(0, i - window_size)
            context = words[start:i]
            # Join context words as a string
            new_texts.append(' '.join(context))
    return new_texts

# Increase context window from 5 to 10
window_size = 10
context_texts = create_context_windows(texts, window_size)

# Since labels correspond to original texts, replicate labels accordingly
expanded_labels = []
for label, text in zip(labels, texts):
    words = text.split()
    expanded_labels.extend([label] * len(words))

# Vectorize text
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(context_texts)
y = np.array(expanded_labels)

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Train logistic regression
model = LogisticRegression(max_iter=200)
model.fit(X_train, y_train)

# Predict and evaluate
train_preds = model.predict(X_train)
val_preds = model.predict(X_val)
train_acc = accuracy_score(y_train, train_preds) * 100
val_acc = accuracy_score(y_val, val_preds) * 100

print(f'Training accuracy: {train_acc:.2f}%')
print(f'Validation accuracy: {val_acc:.2f}%')

Increased the context window size from 5 to 10 words to provide more information to the model.

Adjusted the data preprocessing to create larger context windows accordingly.

Kept the model architecture and dataset unchanged.

Results Interpretation

Before: Training accuracy: 92%, Validation accuracy: 75%

After: Training accuracy: 93%, Validation accuracy: 87%

Increasing the context window size helps the model see more relevant words around the target, improving its understanding and validation accuracy. This reduces overfitting by providing richer input without changing the model itself.

Bonus Experiment

Try using a variable context window size that adapts based on sentence length instead of a fixed size.

💡 Hint

You can create context windows that cover half the sentence length before the target word, which may better capture relevant context for short and long sentences.

Practice

1. What does the term context window mean in natural language processing?

easy

A. A method to remove stop words from text

B. The entire document used for training a model

C. A list of all words in a sentence

D. A small part of text around a word used to understand its meaning

Solution

1. Step 1: Understand the definition of context window

   The context window refers to a limited number of words surrounding a target word to help understand its meaning.

2. Step 2: Compare options with the definition

   Only A small part of text around a word used to understand its meaning correctly describes this as a small part of text around a word. Other options describe unrelated concepts.

3. Final Answer:

   A small part of text around a word used to understand its meaning -> Option D

4. Quick Check:

   Context window = small text part around word [OK]

Hint: Context window = nearby words around a target word [OK]

Common Mistakes:

• Confusing context window with entire document
• Thinking it means all words in a sentence
• Mixing it up with stop word removal

2. Which of the following is the correct way to define a context window of size 3 around the word at index 5 in a list words?

easy

A. words[4:7]

B. words[3:8]

C. words[2:7]

D. words[5:8]

Solution

1. Step 1: Understand context window size and indexing

   A window size of 3 means 3 words total, usually centered on the target word. For index 5, the window covers indices 4, 5, 6.

2. Step 2: Check each option's slice range

   words[4:7] slices words[4:7], which includes indices 4, 5, 6 (3 words). Others include wrong ranges or counts.

3. Final Answer:

   words[4:7] -> Option A

4. Quick Check:

   Window size 3 around index 5 = indices 4 to 6 [OK]

Hint: Slice from index-1 to index+2 for window size 3 [OK]

Common Mistakes:

• Using wrong slice indices causing off-by-one errors
• Including too many or too few words
• Not centering window on target word

3. Given the code below, what will be the output?

words = ['I', 'love', 'to', 'eat', 'apples', 'and', 'bananas']
index = 4
window_size = 3
start = max(0, index - window_size // 2)
end = min(len(words), index + window_size // 2 + 1)
context = words[start:end]
print(context)

medium

A. ['to', 'eat', 'apples']

B. ['eat', 'apples', 'and']

C. ['apples', 'and', 'bananas']

D. ['love', 'to', 'eat']

Solution

1. Step 1: Calculate start and end indices

   window_size is 3, so window_size // 2 = 1. start = max(0, 4 - 1) = 3, end = min(7, 4 + 1 + 1) = 6.

2. Step 2: Extract words from start to end

   words[3:6] = ['eat', 'apples', 'and'].

3. Final Answer:

   ['eat', 'apples', 'and'] -> Option B

4. Quick Check:

   Slice words[3:6] = ['eat', 'apples', 'and'] [OK]

Hint: Calculate start/end with floor division and slice accordingly [OK]

Common Mistakes:

• Off-by-one errors in slicing
• Ignoring max/min boundaries
• Misunderstanding integer division

4. The following code tries to get a context window but sometimes throws an error. What is the main issue?

def get_context(words, index, window_size):
    start = index - window_size // 2
    end = index + window_size // 2 + 1
    return words[start:end]

words = ['hello', 'world']
print(get_context(words, 0, 3))

medium

A. index is out of range

B. window_size must be even

C. start can be negative causing an IndexError

D. The function does not return a list

Solution

1. Step 1: Analyze start index calculation

   For index=0 and window_size=3, start = 0 - 1 = -1, which is negative.

2. Step 2: Understand Python slicing with negative start

   Negative start in slicing accesses from the end, which may cause unexpected results or errors if out of range.

3. Final Answer:

   start can be negative causing an IndexError -> Option C

4. Quick Check:

   Negative start index causes slicing issues [OK]

Hint: Check if start index is negative before slicing [OK]

Common Mistakes:

• Assuming negative indices always work safely
• Thinking window_size must be even
• Ignoring index bounds

5. You want to build a model that uses a context window of size 5 to understand words in sentences. Which approach best handles sentences shorter than 5 words without errors?

hard

A. Pad the sentence with special tokens to length 5 before extracting the window

B. Always extract 5 words ignoring sentence length, causing errors if too short

C. Use only the first word as context if sentence is short

D. Skip sentences shorter than 5 words during training

Solution

1. Step 1: Understand the problem with short sentences

   Sentences shorter than the window size cause indexing errors or incomplete context.

2. Step 2: Evaluate options for handling short sentences

   Padding with special tokens ensures fixed length and avoids errors, unlike skipping or ignoring length.

3. Final Answer:

   Pad the sentence with special tokens to length 5 before extracting the window -> Option A

4. Quick Check:

   Padding fixes short sentence context window issues [OK]

Hint: Pad short sentences to window size to avoid errors [OK]

Common Mistakes:

• Ignoring short sentences causing runtime errors
• Skipping data reduces training quality
• Using incomplete context weakens model understanding