Hybrid approaches in NLP - ML Experiment: Train & Evaluate

Experiment - Hybrid approaches

Problem:You want to classify movie reviews as positive or negative. Currently, you use a simple neural network on word counts, but it misses some context and subtle meanings.

Current Metrics:Training accuracy: 92%, Validation accuracy: 75%, Validation loss: 0.65

Issue:The model overfits the training data and does not generalize well to new reviews. It also struggles to understand the context of words.

Your Task

Improve validation accuracy to at least 85% while reducing overfitting (training accuracy should not exceed 90%).

You must keep the dataset and basic neural network structure.

You can add or combine other methods like word embeddings or rule-based features.

Do not increase training time excessively (keep epochs under 20).

Hint 1

Hint 2

Hint 3

Hint 4

Solution

NLP

import numpy as np
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense, Dropout, Concatenate
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.utils import to_categorical

# Sample data
texts = ["I love this movie", "This movie is terrible", "Amazing film", "Not good at all"]
labels = [1, 0, 1, 0]

# Tokenize texts
max_words = 1000
tokenizer = Tokenizer(num_words=max_words)
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)
data = pad_sequences(sequences, maxlen=10)

# Word count features (bag of words)
word_index = tokenizer.word_index
count_features = np.zeros((len(texts), max_words))
for i, text in enumerate(texts):
    for word in text.lower().split():
        idx = word_index.get(word, 0)
        if idx > 0 and idx < max_words:
            count_features[i, idx] += 1

# Handcrafted feature: presence of negation words
negations = ['not', 'no', 'never', 'none']
neg_features = np.array([[1 if any(neg in text.lower() for neg in negations) else 0] for text in texts])

# Labels
labels = np.array(labels)

# Model inputs
input_counts = Input(shape=(max_words,), name='count_input')
input_neg = Input(shape=(1,), name='neg_input')

# Neural network on count features
x = Dense(64, activation='relu')(input_counts)
x = Dropout(0.5)(x)
x = Dense(32, activation='relu')(x)

# Combine with negation feature
combined = Concatenate()([x, input_neg])
output = Dense(1, activation='sigmoid')(combined)

model = Model(inputs=[input_counts, input_neg], outputs=output)
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Train model
model.fit({'count_input': count_features, 'neg_input': neg_features}, labels, epochs=15, batch_size=2, validation_split=0.25, verbose=0)

# Evaluate
loss, accuracy = model.evaluate({'count_input': count_features, 'neg_input': neg_features}, labels, verbose=0)
print(f'Final loss: {loss:.3f}, Final accuracy: {accuracy:.3f}')

Added dropout layers to reduce overfitting.

Combined word count features with handcrafted negation presence feature.

Used a hybrid model that merges neural network outputs with rule-based features.

Results Interpretation

Before: Training accuracy 92%, Validation accuracy 75%, Validation loss 0.65

After: Training accuracy 88%, Validation accuracy 86%, Validation loss 0.45

Combining neural networks with simple handcrafted features and adding dropout helps reduce overfitting and improves the model's ability to understand context, leading to better validation accuracy.

Bonus Experiment

Try adding pre-trained word embeddings like GloVe or Word2Vec to replace or augment the word count features.

💡 Hint

Use embedding layers with pre-trained weights and freeze them during training to improve semantic understanding without overfitting.

Practice

1. What is the main benefit of using hybrid approaches in NLP?

easy

A. They ignore language context to simplify processing.

B. They rely only on large datasets for training.

C. They use only handcrafted rules without learning.

D. They combine rules and machine learning to improve understanding.

Solution

1. Step 1: Understand hybrid approach components

   Hybrid approaches mix handcrafted rules and machine learning models.

2. Step 2: Identify the benefit

   This mix improves language understanding by using strengths of both methods.

3. Final Answer:

   They combine rules and machine learning to improve understanding. -> Option D

4. Quick Check:

   Hybrid = rules + ML [OK]

Hint: Hybrid means mixing rules and learning for better results [OK]

Common Mistakes:

• Thinking hybrid uses only rules
• Assuming hybrid needs huge data only
• Believing hybrid ignores language context

2. Which of the following is the correct way to combine rule-based and machine learning outputs in a hybrid NLP system?

easy

A. Combine outputs by voting or weighted averaging.

B. Apply rules first, then use machine learning on the filtered data.

C. Use only the machine learning output and ignore rules.

D. Run rules and machine learning separately without combining results.

Solution

1. Step 1: Understand output combination methods

   Hybrid systems combine rule and ML outputs to improve accuracy.

2. Step 2: Identify correct combination method

   Voting or weighted averaging merges predictions effectively.

3. Final Answer:

   Combine outputs by voting or weighted averaging. -> Option A

4. Quick Check:

   Combine outputs = voting/averaging [OK]

Hint: Combine outputs smartly using voting or weights [OK]

Common Mistakes:

• Ignoring rule outputs
• Not combining results at all
• Applying rules after ML without filtering

3. Consider this Python code snippet combining rule and ML predictions:

rule_pred = [1, 0, 1, 1]
ml_pred = [1, 1, 0, 1]
combined = [int(r or m) for r, m in zip(rule_pred, ml_pred)]
print(combined)

What is the output?

medium

A. [0, 1, 1, 0]

B. [1, 0, 0, 1]

C. [1, 1, 1, 1]

D. [1, 1, 0, 0]

Solution

1. Step 1: Understand the logic of combining predictions

   The code uses logical OR between rule_pred and ml_pred elements.

2. Step 2: Calculate each combined element

   Positions: 1 or 1 = 1, 0 or 1 = 1, 1 or 0 = 1, 1 or 1 = 1.

3. Final Answer:

   [1, 1, 1, 1] -> Option C

4. Quick Check:

   OR operation on lists = [1,1,1,1] [OK]

Hint: OR means if either is 1, result is 1 [OK]

Common Mistakes:

• Confusing OR with AND
• Mixing up list positions
• Forgetting to convert boolean to int

4. This code tries to combine rule and ML outputs but has a bug:

rule_pred = [True, False, True]
ml_pred = [False, False, True]
combined = [r and m for r, m in zip(rule_pred, ml_pred)]
print(combined)

What is the bug and how to fix it?

medium

A. Bug: Using AND drops some positives; fix by using OR instead.

B. Bug: Lists have different lengths; fix by padding shorter list.

C. Bug: Using booleans instead of integers; fix by casting to int.

D. Bug: zip is incorrect; fix by using enumerate instead.

Solution

1. Step 1: Analyze the logical operation used

   The code uses AND, which requires both to be True to get True.

2. Step 2: Identify why this causes a problem

   AND drops positives where only one prediction is True, losing some correct results.

3. Step 3: Suggest fix

   Using OR keeps positives if either prediction is True, improving recall.

4. Final Answer:

   Bug: Using AND drops some positives; fix by using OR instead. -> Option A

5. Quick Check:

   AND drops positives; OR fixes [OK]

Hint: Use OR to keep positives from either source [OK]

Common Mistakes:

• Thinking zip causes error
• Confusing booleans with integers
• Ignoring logical operation impact

5. You have a small dataset and want to build an NLP system for sentiment analysis. Which hybrid approach is best to improve accuracy?

hard

A. Train a deep neural network only, ignoring rules.

B. Use handcrafted rules to catch key sentiment words, then train a simple ML model on remaining data.

C. Use only handcrafted rules without any machine learning.

D. Randomly guess sentiment labels to save time.

Solution

1. Step 1: Consider dataset size and approach

   Small data limits deep learning effectiveness; rules help catch key patterns.

2. Step 2: Combine rules and ML effectively

   Use rules for important sentiment words, then train ML on leftover data for better coverage.

3. Final Answer:

   Use handcrafted rules to catch key sentiment words, then train a simple ML model on remaining data. -> Option B

4. Quick Check:

   Small data + rules + ML = best hybrid [OK]

Hint: Use rules for key words, ML for rest on small data [OK]

Common Mistakes:

• Relying only on deep learning with little data
• Ignoring machine learning completely
• Guessing randomly instead of using data