Beam search decoding in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Beam search decoding

Which metric matters for Beam Search Decoding and WHY

Beam search is used in tasks like language translation or text generation. The main goal is to find the best sequence of words. So, metrics that measure how good the generated sequences are matter most. These include BLEU score (how close the output is to a reference), perplexity (how well the model predicts the next word), and sequence accuracy (exact match rate). These metrics tell us if beam search helps find better sentences than simpler methods.

Confusion Matrix or Equivalent Visualization

Beam search decoding does not produce a simple confusion matrix like classification. Instead, we can visualize the beam paths and their scores. For example, if beam width is 3, at each step the top 3 partial sequences are kept. The scores show how likely each sequence is. A simple ASCII example for one step:

Step 1: ["I" (score=0.9), "It" (0.7), "In" (0.5)]
Step 2: 
  "I am" (0.8), "I is" (0.6), "I was" (0.5)
  "It is" (0.7), "It was" (0.4), "It can" (0.3)
  "In the" (0.5), "In a" (0.4), "In my" (0.3)
Top 3 sequences kept based on scores.
    

This shows how beam search narrows down the best sequences step by step.

Precision vs Recall Tradeoff (or Equivalent) with Concrete Examples

Beam search balances between exploring many possible sequences (high recall of options) and focusing on the best ones (high precision). A small beam width means fewer sequences kept, so it is fast but might miss the best sentence (low recall). A large beam width keeps many sequences, increasing chance to find the best output (high recall), but is slower and may include many poor sequences (lower precision).

Example: In machine translation, a beam width of 1 (greedy search) might miss a better translation. A beam width of 10 finds better sentences but takes more time. Choosing beam width is a tradeoff between quality and speed.

What "Good" vs "Bad" Metric Values Look Like for Beam Search Decoding

Good: High BLEU score (close to human translation), low perplexity, and high sequence accuracy. This means beam search finds sequences that match references well and the model predicts words confidently.

Bad: Low BLEU score, high perplexity, or low sequence accuracy. This means beam search is not helping or is stuck in poor sequences. For example, if increasing beam width does not improve BLEU, the search might be ineffective.

Common Metrics Pitfalls

• Ignoring diversity: Beam search can produce very similar sequences, so metrics might look good but outputs lack variety.
• Overfitting to references: BLEU score depends on reference sentences; good BLEU does not always mean better meaning.
• Beam width too small or too large: Too small misses good sequences; too large wastes time and may pick worse sequences due to score biases.
• Not considering length bias: Beam search can favor shorter or longer sequences unfairly, affecting metric scores.

Self Check

Your beam search model has a BLEU score of 25 but increasing beam width from 5 to 20 does not improve BLEU. Is your beam search working well? Why or why not?

Answer: The beam search might not be effective beyond width 5. It could be stuck in local optima or the model scores do not help find better sequences. Increasing beam width wastes time without quality gain. You may need to check model scoring or try other decoding methods.

Key Result

Beam search decoding quality is best measured by sequence-level metrics like BLEU and perplexity, balancing beam width for quality and speed.