BERT for text classification in PyTorch

Introduction

BERT helps computers understand text better by reading words in context. We use it to teach a model to decide what category a piece of text belongs to.

Sorting emails into categories like spam or important

Classifying customer reviews as positive or negative

Detecting the topic of news articles automatically

Filtering social media posts by sentiment or subject

Organizing support tickets by issue type

Syntax

PyTorch

from transformers import BertTokenizer, BertForSequenceClassification
import torch

# Load tokenizer and model
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)

# Prepare input text
inputs = tokenizer('Your text here', return_tensors='pt', padding=True, truncation=True)

# Get model output
outputs = model(**inputs)

# Extract logits for classification
logits = outputs.logits

Use BertTokenizer to convert text into numbers the model understands.

BertForSequenceClassification is a BERT model with a classification head on top.

Examples

Classify a positive sentence.

PyTorch

inputs = tokenizer('I love this product!', return_tensors='pt', padding=True, truncation=True)
outputs = model(**inputs)
logits = outputs.logits

Classify multiple sentences at once with padding.

PyTorch

inputs = tokenizer(['Good', 'Bad'], return_tensors='pt', padding=True, truncation=True)
outputs = model(**inputs)
logits = outputs.logits

Sample Model

This code classifies two sentences as positive or negative using BERT. It shows the loss and predicted labels.

PyTorch

from transformers import BertTokenizer, BertForSequenceClassification
import torch
from torch.nn.functional import cross_entropy

# Sample data: texts and labels
texts = ['I love this movie', 'This film is terrible']
labels = torch.tensor([1, 0])  # 1=positive, 0=negative

# Load tokenizer and model
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)

# Prepare inputs
inputs = tokenizer(texts, return_tensors='pt', padding=True, truncation=True)

# Forward pass
outputs = model(**inputs)
logits = outputs.logits

# Calculate loss
loss = cross_entropy(logits, labels)

# Predictions
predictions = torch.argmax(logits, dim=1)

print(f'Loss: {loss.item():.4f}')
print(f'Predictions: {predictions.tolist()}')

Important Notes

BERT models are large; running on a GPU speeds up training and prediction.

Always tokenize and pad inputs to the same length for batch processing.

Output logits are raw scores; use softmax or argmax to get predicted classes.

Summary

BERT reads text with context to improve classification accuracy.

Use BertTokenizer and BertForSequenceClassification to build text classifiers easily.

Prepare inputs carefully and interpret logits to get predictions.