Dependency parsing in NLP - ML Experiment: Train & Evaluate

Experiment - Dependency parsing

Problem:We want to build a model that can analyze sentences and find the grammatical relationships between words, called dependency parsing. The current model is trained on a small dataset and achieves 95% accuracy on training data but only 70% accuracy on validation data.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 0.45

Issue:The model is overfitting: it performs very well on training data but poorly on validation data.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85%, while keeping training accuracy below 92%.

You can only change model hyperparameters and add regularization techniques.

Do not change the dataset or model architecture drastically.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

NLP

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping

# Sample data placeholders (replace with actual data loading)
X_train, y_train = ...  # training data
X_val, y_val = ...      # validation data

vocab_size = 10000
embedding_dim = 128
max_length = 50

model = Sequential([
    Embedding(vocab_size, embedding_dim, input_length=max_length),
    LSTM(64, return_sequences=True),
    Dropout(0.5),
    LSTM(32),
    Dropout(0.5),
    Dense(32, activation='relu'),
    Dense(vocab_size, activation='softmax')
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

early_stop = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

history = model.fit(X_train, y_train,
                    epochs=20,
                    batch_size=64,
                    validation_data=(X_val, y_val),
                    callbacks=[early_stop])

Added Dropout layers after LSTM layers with rate 0.5 to reduce overfitting.

Reduced LSTM units from 128 to 64 and 32 to lower model complexity.

Added EarlyStopping callback to stop training when validation loss stops improving.

Set learning rate to 0.001 for stable training.

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 70%, Training loss 0.15, Validation loss 0.45

After: Training accuracy 90%, Validation accuracy 87%, Training loss 0.25, Validation loss 0.30

Adding dropout and early stopping helped reduce overfitting, improving validation accuracy while slightly lowering training accuracy. This shows how regularization and training control help models generalize better.

Bonus Experiment

Try using a pretrained language model like BERT for dependency parsing and compare the results.

💡 Hint

Use transfer learning with a pretrained BERT model and fine-tune it on your dependency parsing dataset.