Hugging Face integration basics in PyTorch - ML Experiment: Train & Evaluate

Experiment - Hugging Face integration basics

Problem:You want to fine-tune a Hugging Face transformer model for text classification but your model overfits. Training accuracy is 98% but validation accuracy is only 70%.

Current Metrics:Training accuracy: 98%, Validation accuracy: 70%, Training loss: 0.05, Validation loss: 0.8

Issue:The model is overfitting the training data and does not generalize well to validation data.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85% while keeping training accuracy below 92%.

You can only modify the model training code (no changes to dataset).

Use PyTorch and Hugging Face transformers.

Keep the same transformer architecture.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

PyTorch

import torch
from torch.utils.data import DataLoader
from transformers import AutoModelForSequenceClassification, AutoTokenizer, AutoConfig, AdamW, get_scheduler
from datasets import load_dataset

# Load dataset
raw_datasets = load_dataset('imdb')

# Load tokenizer and model
model_name = 'distilbert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(model_name)
config = AutoConfig.from_pretrained(model_name)
config.hidden_dropout_prob = 0.3
config.attention_probs_dropout_prob = 0.3
model = AutoModelForSequenceClassification.from_pretrained(model_name, config=config, num_labels=2)

# Tokenize function
def tokenize_function(examples):
    tokenized = tokenizer(examples['text'], padding='max_length', truncation=True, max_length=128)
    tokenized['labels'] = examples['label']
    return tokenized

# Tokenize datasets
encoded_datasets = raw_datasets.map(tokenize_function, batched=True)

# Prepare dataloaders
train_dataset = encoded_datasets['train'].shuffle(seed=42).select(range(2000))
val_dataset = encoded_datasets['test'].shuffle(seed=42).select(range(500))

train_loader = DataLoader(train_dataset, batch_size=16, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=16)

# Optimizer and scheduler
optimizer = AdamW(model.parameters(), lr=5e-5, weight_decay=0.01)
num_epochs = 3
num_training_steps = num_epochs * len(train_loader)
scheduler = get_scheduler('linear', optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps)

# Training loop with early stopping
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model.to(device)

best_val_acc = 0
patience = 2
patience_counter = 0

for epoch in range(num_epochs):
    model.train()
    for batch in train_loader:
        batch = {k: v.to(device) for k, v in batch.items() if k in ['input_ids', 'attention_mask', 'labels']}
        outputs = model(**batch)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        scheduler.step()
        optimizer.zero_grad()

    # Validation
    model.eval()
    correct = 0
    total = 0
    with torch.no_grad():
        for batch in val_loader:
            batch = {k: v.to(device) for k, v in batch.items() if k in ['input_ids', 'attention_mask', 'labels']}
            outputs = model(**batch)
            predictions = outputs.logits.argmax(dim=-1)
            correct += (predictions == batch['labels']).sum().item()
            total += batch['labels'].size(0)
    val_acc = correct / total * 100
    print(f'Epoch {epoch+1}: Validation Accuracy: {val_acc:.2f}%')

    # Early stopping
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        patience_counter = 0
    else:
        patience_counter += 1
        if patience_counter >= patience:
            print('Early stopping triggered')
            break

Added dropout rate of 0.3 in the transformer model to reduce overfitting.

Reduced batch size to 16 for better generalization.

Added weight decay (0.01) to optimizer to regularize weights.

Implemented learning rate scheduler with linear decay.

Added early stopping with patience of 2 epochs to stop training when validation accuracy stops improving.

Results Interpretation

Before: Training accuracy 98%, Validation accuracy 70%, Training loss 0.05, Validation loss 0.8

After: Training accuracy 90%, Validation accuracy 87%, Training loss 0.25, Validation loss 0.35

Adding dropout, weight decay, and early stopping helps reduce overfitting. This improves validation accuracy by making the model generalize better, even if training accuracy decreases slightly.

Bonus Experiment

Try using data augmentation techniques like back translation or synonym replacement to further improve validation accuracy.

💡 Hint

Augmenting text data can increase diversity and help the model learn more robust features.

Practice

1. What is the main purpose of the AutoTokenizer class in Hugging Face integration with PyTorch?

easy

A. To visualize model predictions

B. To convert raw text into tokens that the model can understand

C. To train a new language model from scratch

D. To save the model weights to disk

Solution

1. Step 1: Understand the role of tokenization

   Tokenization breaks down raw text into smaller pieces called tokens, which models use as input.

2. Step 2: Identify what AutoTokenizer does

   AutoTokenizer automatically loads the correct tokenizer for a given pretrained model to convert text into tokens.

3. Final Answer:

   To convert raw text into tokens that the model can understand -> Option B

4. Quick Check:

   AutoTokenizer = text to tokens [OK]

Hint: Tokenizer = text to tokens for model input [OK]

Common Mistakes:

• Confusing tokenizer with model training
• Thinking tokenizer saves model weights
• Assuming tokenizer visualizes outputs

2. Which of the following is the correct way to load a pretrained model using Hugging Face's AutoModel in PyTorch?

easy

A. model = AutoModel.from_pretrained('bert-base-uncased')

B. model = AutoModel.load('bert-base-uncased')

C. model = AutoModel('bert-base-uncased')

D. model = AutoModel.pretrained('bert-base-uncased')

Solution

1. Step 1: Recall the correct method to load pretrained models

   Hugging Face uses the from_pretrained method to load pretrained weights and configuration.

2. Step 2: Match the correct syntax

   The correct syntax is AutoModel.from_pretrained('model_name').

3. Final Answer:

   model = AutoModel.from_pretrained('bert-base-uncased') -> Option A

4. Quick Check:

   Load model with from_pretrained() [OK]

Hint: Use from_pretrained() to load models [OK]

Common Mistakes:

• Using .load() instead of .from_pretrained()
• Calling AutoModel like a function
• Using non-existent .pretrained() method

3. Given the following code snippet, what will be the shape of outputs.last_hidden_state?

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModel.from_pretrained('bert-base-uncased')

inputs = tokenizer('Hello world!', return_tensors='pt')
outputs = model(**inputs)

medium

A. [2, 768]

B. [1, 768]

C. [1, 5, 768]

D. [768, 2]

Solution

1. Step 1: Understand the tokenizer output

   The tokenizer returns input IDs with shape [batch_size, sequence_length]. Here batch_size=1 and sequence_length=5 (tokens: [CLS], 'hello', 'world', '!', [SEP]).

2. Step 2: Understand model output shape

   The model outputs last_hidden_state with shape [batch_size, sequence_length, hidden_size]. For BERT base, hidden_size=768.

3. Final Answer:

   [1, 5, 768] -> Option C

4. Quick Check:

   Output shape = [batch, tokens, features] = [1, 5, 768] [OK]

Hint: Output shape = batch x tokens x hidden size [OK]

Common Mistakes:

• Confusing sequence length with batch size
• Ignoring hidden size dimension
• Assuming output shape is 2D instead of 3D

4. Identify the error in this Hugging Face PyTorch code snippet:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModel.from_pretrained('bert-base-uncased')

inputs = tokenizer('Test input', return_tensors='pt')
outputs = model(inputs)
print(outputs)

medium

A. Missing .to(device) to move inputs to GPU

B. AutoModel cannot be used with BERT models

C. Tokenizer should use return_tensors='tf' instead of 'pt'

D. Model should be called with unpacked inputs: model(**inputs)

Solution

1. Step 1: Check how model is called

   The model expects keyword arguments, so inputs must be unpacked with ** to pass input_ids and attention_mask correctly.

2. Step 2: Identify the fix

   Change model(inputs) to model(**inputs) to avoid errors.

3. Final Answer:

   Model should be called with unpacked inputs: model(**inputs) -> Option D

4. Quick Check:

   Use **inputs when calling model [OK]

Hint: Call model with model(**inputs) not model(inputs) [OK]

Common Mistakes:

• Passing inputs as a single dict argument
• Confusing PyTorch and TensorFlow tensor types
• Thinking AutoModel can't load BERT

5. You want to classify text into positive or negative sentiment using a Hugging Face model in PyTorch. After tokenizing and running the model, which step correctly converts raw model outputs into class probabilities?

hard

A. Apply torch.softmax on the model's logits output

B. Use torch.argmax directly on tokenized inputs

C. Call model.eval() to get probabilities

D. Normalize input tokens before feeding to the model

Solution

1. Step 1: Understand model output for classification

   Models output raw scores called logits for each class, which are not probabilities.

2. Step 2: Convert logits to probabilities

   Applying torch.softmax on logits converts them into probabilities summing to 1.

3. Final Answer:

   Apply torch.softmax on the model's logits output -> Option A

4. Quick Check:

   Softmax = logits to probabilities [OK]

Hint: Use softmax on logits to get probabilities [OK]

Common Mistakes:

• Using argmax on inputs instead of outputs
• Confusing model.eval() with probability conversion
• Normalizing inputs instead of outputs