Sequence-to-sequence architecture in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Sequence-to-sequence architecture

Which metric matters for Sequence-to-sequence architecture and WHY

Sequence-to-sequence models create one sequence from another, like translating languages or summarizing text. The main metrics to check are BLEU and ROUGE. These compare the model's output to the correct answer by looking at matching words or phrases. BLEU focuses on precision (how many predicted words are correct), while ROUGE focuses on recall (how many correct words were found). These metrics matter because they show how well the model captures the meaning and structure of the target sequence.

Confusion matrix or equivalent visualization

For sequence-to-sequence tasks, confusion matrices are less common because outputs are sequences, not single labels. Instead, we use n-gram overlap counts. For example, BLEU counts how many 1-word, 2-word, 3-word, and 4-word sequences in the prediction match the reference.

Reference:  "I love machine learning"
Prediction: "I enjoy machine learning"

1-gram matches: I, machine, learning (3 matches)
2-gram matches: machine learning (1 match)
3-gram matches: none
4-gram matches: none
    

This overlap helps calculate BLEU or ROUGE scores.

Precision vs Recall tradeoff with concrete examples

In sequence-to-sequence, precision means how many predicted words are correct, recall means how many correct words were predicted.

Example 1: High precision, low recall
Model predicts only very common words it is sure about, so most predicted words are correct but misses many words from the reference. Result: output is short and incomplete.

Example 2: High recall, low precision
Model predicts many words including many incorrect ones. It covers most of the reference words but adds noise. Result: output is long but less accurate.

Good models balance precision and recall to produce fluent and accurate sequences.

What "good" vs "bad" metric values look like for this use case

Good BLEU/ROUGE scores: Around 0.5 to 0.7 or higher usually means the model produces meaningful and relevant sequences close to the reference.

Bad BLEU/ROUGE scores: Below 0.2 means the model output is often very different from the reference, likely missing key words or structure.

Note: Scores depend on task difficulty and dataset size, but higher is always better.

Metrics pitfalls

• Ignoring sequence length: Very short outputs can get high precision but miss meaning.
• Overfitting: Model memorizes training sequences, scoring high on training but low on new data.
• Data leakage: If test data is too similar to training, metrics look better than real performance.
• BLEU limitations: It does not measure meaning or grammar well, only word overlap.
• ROUGE limitations: Focuses on recall, so it can favor longer outputs with extra words.

Self-check question

Your sequence-to-sequence model has a BLEU score of 0.65 on the test set but produces very short summaries missing important details. Is this good?

Answer: Not fully. A high BLEU score is good, but very short outputs missing details show the model may have high precision but low recall. You want balanced metrics and qualitative checks to ensure summaries are complete and useful.

Key Result

BLEU and ROUGE scores measure how well sequence-to-sequence models match target sequences by comparing word overlaps, balancing precision and recall.

Practice

1. What is the main role of the encoder in a sequence-to-sequence model?

easy

A. To generate the output sequence directly

B. To read and understand the input sequence

C. To evaluate the model's accuracy

D. To preprocess the data before training

Solution

1. Step 1: Understand the encoder's function

   The encoder processes the input sequence and converts it into a meaningful representation.

2. Step 2: Differentiate encoder from decoder

   The decoder uses this representation to generate the output sequence, so it does not directly read input.

3. Final Answer:

   To read and understand the input sequence -> Option B

4. Quick Check:

   Encoder = input reader [OK]

Hint: Encoder reads input; decoder writes output [OK]

Common Mistakes:

• Confusing encoder with decoder
• Thinking encoder generates output
• Assuming encoder evaluates accuracy

2. Which of the following is the correct way to describe the decoder in a sequence-to-sequence model?

easy

A. It generates the output sequence from the encoded input

B. It encodes the input sequence into a fixed vector

C. It normalizes the input data before encoding

D. It splits the input sequence into smaller parts

Solution

1. Step 1: Identify decoder's role

   The decoder takes the encoded input and produces the output sequence step-by-step.

2. Step 2: Eliminate incorrect options

   Encoding is done by the encoder, not the decoder; normalization and splitting are preprocessing steps.

3. Final Answer:

   It generates the output sequence from the encoded input -> Option A

4. Quick Check:

   Decoder = output generator [OK]

Hint: Decoder creates output from encoder's info [OK]

Common Mistakes:

• Mixing encoder and decoder roles
• Confusing preprocessing with decoding
• Assuming decoder encodes input

3. Consider this simplified pseudocode for a sequence-to-sequence model:

encoded = encoder(input_sequence)
output = decoder(encoded)
print(len(output))

If the input sequence length is 5 and the model is trained to translate to a sequence of length 7, what will len(output) print?

medium

A. 5

B. Cannot determine without more info

C. 12

D. 7

Solution

1. Step 1: Understand input and output lengths

   The input sequence length is 5, but the model is trained to produce output sequences of length 7.

2. Step 2: Recognize decoder output length

   The decoder generates output sequences based on training, so output length should be 7 regardless of input length.

3. Final Answer:

   7 -> Option D

4. Quick Check:

   Output length = trained target length = 7 [OK]

Hint: Output length matches target, not input length [OK]

Common Mistakes:

• Assuming output length equals input length
• Adding input and output lengths
• Saying output length is unknown

4. You have this code snippet for a sequence-to-sequence model training step:

for input_seq, target_seq in dataset:
    encoded = encoder(input_seq)
    output = decoder(encoded)
    loss = loss_function(output, target_seq)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

What is the likely error in this code?

medium

A. optimizer.zero_grad() should be called before loss.backward()

B. optimizer.step() should be called before loss.backward()

C. Missing call to optimizer.zero_grad() before loss.backward()

D. optimizer.zero_grad() should be called before optimizer.step()

Solution

1. Step 1: Recall training step order

   Gradients must be cleared before computing new gradients with loss.backward().

2. Step 2: Identify correct zero_grad() placement

   optimizer.zero_grad() should be called before loss.backward(), not after optimizer.step().

3. Final Answer:

   Missing call to optimizer.zero_grad() before loss.backward() -> Option C

4. Quick Check:

   Clear grads before backward pass [OK]

Hint: Call zero_grad() before backward() [OK]

Common Mistakes:

• Calling zero_grad() after backward()
• Calling optimizer.step() before backward()
• Skipping zero_grad() entirely

5. In a sequence-to-sequence model for language translation, why might adding an attention mechanism improve performance?

hard

A. It allows the decoder to focus on relevant parts of the input sequence dynamically

B. It reduces the size of the input sequence to a fixed vector

C. It speeds up training by skipping the encoder step

D. It replaces the decoder with a simpler model

Solution

1. Step 1: Understand attention's purpose

   Attention helps the decoder look at different parts of the input sequence when generating each output token.

2. Step 2: Compare with fixed vector encoding

   Without attention, the encoder compresses input into one fixed vector, which can lose details.

3. Step 3: Eliminate incorrect options

   Attention does not reduce input size, skip encoder, or replace decoder; it enhances focus during decoding.

4. Final Answer:

   It allows the decoder to focus on relevant parts of the input sequence dynamically -> Option A

5. Quick Check:

   Attention = dynamic focus on input [OK]

Hint: Attention helps decoder focus on input parts [OK]

Common Mistakes:

• Thinking attention reduces input size
• Believing attention skips encoder
• Assuming attention replaces decoder