Encoder-decoder with attention in NLP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main purpose of the encoder in an encoder-decoder model?

The encoder reads the input data and converts it into a fixed-size representation (a context vector) that summarizes the input information for the decoder to use.

beginner

Why do we use attention in encoder-decoder models?

Attention helps the decoder focus on different parts of the input sequence at each step, instead of relying on a single fixed context vector. This improves performance, especially for long sequences.

intermediate

Describe how the attention mechanism works in simple terms.

At each step, the decoder looks at all encoder outputs and assigns weights (attention scores) to them. These weights show how important each input part is for generating the current output word.

intermediate

What is the difference between the context vector in a basic encoder-decoder and one with attention?

In a basic model, the context vector is fixed and the same for all output steps. With attention, the context vector changes at each step, computed as a weighted sum of encoder outputs based on attention scores.

intermediate

How does attention improve translation quality in machine translation tasks?

Attention allows the model to align output words with relevant input words dynamically, helping it handle long sentences and complex structures better than fixed context models.

What does the encoder output in an encoder-decoder model with attention?

A The final output sentence

B A single word prediction

C A sequence of hidden states representing the input

D The loss value

The encoder outputs a sequence of hidden states that represent the input sequence, which the attention mechanism uses.

In attention, what do the attention weights represent?

A The length of the input sequence

B The importance of each input token for the current output token

C The number of layers in the model

D The learning rate

Attention weights show how much each input token should influence the current output token.

Why is a fixed context vector limiting in basic encoder-decoder models?

A It cannot capture all input details for long sequences

B It increases training speed

C It reduces model size

D It improves output diversity

A fixed context vector compresses all input information into one vector, which can lose details especially for long inputs.

Which part of the model uses attention scores to generate output?

A Decoder

B Encoder

C Input layer

D Loss function

The decoder uses attention scores to focus on relevant encoder outputs when producing each output token.

What is a common benefit of adding attention to encoder-decoder models?

A Less training data needed

B Faster inference without accuracy change

C Simpler model architecture

D Better handling of long input sequences

Attention helps models better process long sequences by focusing on important parts dynamically.

Explain how the attention mechanism changes the way the decoder generates each output token compared to a basic encoder-decoder model.

Think about how the decoder decides what input information to use at each step.

Describe the roles of the encoder, decoder, and attention mechanism in an encoder-decoder model with attention.

Consider how these components work together to produce output.