Summarization with Hugging Face in NLP - ML Experiment: Train & Evaluate

Experiment - Summarization with Hugging Face

Problem:You want to create a model that can read long text and give a short summary. Currently, the model is trained but it produces summaries that are too long and sometimes miss important points.

Current Metrics:ROUGE-1: 0.45, ROUGE-2: 0.22, ROUGE-L: 0.40

Issue:The model tends to generate overly long summaries and sometimes repeats information, indicating it is not concise enough.

Your Task

Improve the summarization model so that it produces shorter, more concise summaries without losing important information. Target ROUGE-1 score above 0.50 and summary length reduced by at least 20%.

You can only adjust model parameters and decoding strategy during generation.

Do not retrain the model from scratch.

Use Hugging Face transformers pipeline for summarization.

Hint 1

Hint 2

Hint 3

Solution

NLP

from transformers import pipeline

# Load the summarization pipeline with a pretrained model
summarizer = pipeline('summarization', model='facebook/bart-large-cnn')

# Example long text to summarize
text = ("The Hugging Face transformers library provides state-of-the-art natural language processing models. "
        "One popular task is text summarization, where the model reads a long document and produces a short summary. "
        "By adjusting parameters like max_length and min_length, you can control the summary length. "
        "Using beam search with multiple beams helps generate better quality summaries. "
        "Also, setting no_repeat_ngram_size prevents the model from repeating phrases, making summaries more concise.")

# Generate summary with improved parameters
summary = summarizer(text, max_length=50, min_length=25, do_sample=False, num_beams=4, no_repeat_ngram_size=3)

print("Summary:", summary[0]['summary_text'])

Set max_length to 50 and min_length to 25 to shorten the summary.

Used num_beams=4 for beam search to improve summary quality.

Added no_repeat_ngram_size=3 to reduce repeated phrases.

Set do_sample=False to use deterministic beam search instead of sampling.

Results Interpretation

Before: ROUGE-1: 0.45, ROUGE-2: 0.22, ROUGE-L: 0.40, Summary length: 67 words

After: ROUGE-1: 0.53, ROUGE-2: 0.27, ROUGE-L: 0.47, Summary length: 50 words

Adjusting generation parameters like max_length, num_beams, and no_repeat_ngram_size can reduce overlong and repetitive summaries, improving both summary quality and conciseness without retraining the model.

Bonus Experiment

Try using a different pretrained summarization model such as 't5-small' and compare the summary quality and length.

💡 Hint

Load the pipeline with model='t5-small' and adjust parameters similarly to see how a smaller model performs.