RNN-based text generation in NLP - Model Metrics & Evaluation

Metrics & Evaluation - RNN-based text generation

Which metric matters for RNN-based text generation and WHY

For RNN text generation, the main goal is to produce text that looks natural and meaningful. We often use perplexity to measure this. Perplexity tells us how well the model predicts the next word. A lower perplexity means the model is better at guessing the next word, so the generated text is more fluent.

Sometimes, we also check BLEU score if we have reference texts to compare. BLEU measures how similar the generated text is to real examples. But perplexity is the most common because it works even without exact references.

Confusion matrix or equivalent visualization

In text generation, we don't use a confusion matrix like in classification. Instead, we look at perplexity, which is calculated from the probabilities the model assigns to the correct next words.

Perplexity = exp(- (1/N) * sum(log P(w_i | context)))

Where:
- N is the number of words in the test set
- P(w_i | context) is the predicted probability of the actual next word

Lower perplexity means better prediction.
    

Precision vs Recall tradeoff with concrete examples

Precision and recall are not typical for text generation. Instead, we think about a tradeoff between creativity and coherence.

If the model is too safe (high coherence), it repeats common phrases and is boring. This is like high precision but low recall -- it only generates very safe words.

If the model is too creative (low coherence), it may produce strange or wrong words. This is like high recall but low precision -- it tries many words but many are bad.

Good text generation balances this tradeoff, producing text that is both interesting and makes sense.

What "good" vs "bad" metric values look like for RNN text generation

Good perplexity: Lower values, often between 20 and 50 for typical datasets, mean the model predicts next words well.

Bad perplexity: Very high values (100+) mean the model struggles to predict next words, so generated text is often nonsensical.

For BLEU (if used), scores closer to 1.0 mean generated text matches references well; scores near 0 mean poor match.

Common pitfalls in metrics for RNN text generation

• Overfitting: Very low perplexity on training data but high on test data means the model memorizes text and won't generalize.
• Ignoring diversity: Low perplexity alone doesn't guarantee interesting text; the model might repeat the same phrases.
• Using BLEU without references: BLEU needs reference texts; without them, it's not useful.
• Perplexity scale: Perplexity depends on vocabulary size and dataset; comparing across different setups can be misleading.

Self-check question

Your RNN text generation model has a perplexity of 25 on training data but 120 on test data. Is it good for generating natural text? Why or why not?

Answer: No, this is not good. The model performs well on training data but poorly on test data, showing it overfits. It memorizes training text but cannot generalize to new text, so generated text will likely be poor and unnatural.

Key Result

Perplexity is key: lower perplexity means the RNN predicts next words better, producing more natural text.

Practice

1. What is the main purpose of using an RNN in text generation?

easy

A. To count the number of words in a sentence

B. To sort words alphabetically

C. To translate text into another language

D. To learn patterns in sequences of words to predict the next word

Solution

1. Step 1: Understand RNN function in text

   RNNs process sequences step-by-step, remembering past words to predict what comes next.

2. Step 2: Identify the goal of text generation

   The goal is to generate new text by predicting the next word based on learned patterns.

3. Final Answer:

   To learn patterns in sequences of words to predict the next word -> Option D

4. Quick Check:

   RNN predicts next word in sequence = C [OK]

Hint: RNNs remember word order to guess the next word [OK]

Common Mistakes:

• Thinking RNNs just count words
• Confusing RNNs with sorting algorithms
• Assuming RNNs translate text directly

2. Which of the following is the correct way to define an embedding layer in a Keras RNN model for text generation?

easy

A. Embedding(input_length=64, input_dim=10, output_dim=1000)

B. Embedding(output_dim=1000, input_dim=64, input_length=10)

C. Embedding(input_dim=1000, output_dim=64, input_length=10)

D. Embedding(input_dim=10, output_dim=1000, input_length=64)

Solution

1. Step 1: Recall embedding layer parameters

   Embedding layers require input_dim (vocab size), output_dim (embedding size), and input_length (sequence length).

2. Step 2: Match parameters correctly

   Embedding(input_dim=1000, output_dim=64, input_length=10) correctly sets input_dim=1000 (vocab size), output_dim=64 (embedding size), input_length=10 (sequence length).

3. Final Answer:

   Embedding(input_dim=1000, output_dim=64, input_length=10) -> Option C

4. Quick Check:

   Embedding(input_dim, output_dim, input_length) = A [OK]

Hint: Input_dim = vocab size, output_dim = embedding size [OK]

Common Mistakes:

• Swapping input_dim and output_dim
• Confusing input_length with output_dim
• Using wrong parameter names

3. Given this code snippet for training an RNN text generator, what will be the shape of the input data X if the vocabulary size is 5000, sequence length is 20, and batch size is 32?

model = Sequential()
model.add(Embedding(input_dim=5000, output_dim=50, input_length=20))
model.add(SimpleRNN(100))
model.add(Dense(5000, activation='softmax'))

X = np.random.randint(0, 5000, (32, 20))

medium

A. (20, 32)

B. (32, 20)

C. (32, 50)

D. (5000, 20)

Solution

1. Step 1: Understand input shape for embedding

   The input to the embedding layer is a 2D array: (batch_size, sequence_length).

2. Step 2: Check given data shape

   X is created with shape (32, 20), matching batch size 32 and sequence length 20.

3. Final Answer:

   (32, 20) -> Option B

4. Quick Check:

   Input shape = (batch_size, sequence_length) = (32, 20) [OK]

Hint: Input shape = batch size by sequence length [OK]

Common Mistakes:

• Confusing embedding output shape with input shape
• Swapping batch size and sequence length
• Assuming embedding changes input shape

4. You wrote this code to train an RNN text generator but get a shape mismatch error:

model = Sequential()
model.add(Embedding(input_dim=10000, output_dim=64, input_length=15))
model.add(SimpleRNN(128))
model.add(Dense(10000, activation='softmax'))

X = np.random.randint(0, 10000, (64, 15))
y = np.random.randint(0, 10000, (64, 15))  # target labels

model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')
model.fit(X, y, epochs=5)

What is the main issue causing the error?

medium

A. Target labels y should be shape (64,) with integer word indices, not (64, 15)

B. Embedding input_dim is too large

C. SimpleRNN units should match output_dim of embedding

D. Loss function sparse_categorical_crossentropy is incorrect

Solution

1. Step 1: Check target label shape for next word prediction

   For next word prediction, y should be a 1D array of word indices (batch_size,), not sequences.

2. Step 2: Identify mismatch in y shape

   y has shape (64, 15), which causes shape mismatch with model output (64, 10000).

3. Final Answer:

   Target labels y should be shape (64,) with integer word indices, not (64, 15) -> Option A

4. Quick Check:

   y shape must match output shape = B [OK]

Hint: Targets for next word are 1D, not sequences [OK]

Common Mistakes:

• Using sequences as targets instead of next word
• Confusing embedding size with RNN units
• Changing loss function unnecessarily

5. You want to generate text using a trained RNN model. Which approach correctly generates text word by word after training?

hard

A. Feed the model the initial seed sequence, predict the next word, append it, then use the updated sequence to predict again

B. Feed the entire training dataset at once to get all generated words

C. Use the model to predict all words simultaneously without updating input

D. Randomly select words from the vocabulary without using the model

Solution

1. Step 1: Understand sequential generation

   Text generation uses the model to predict one word at a time, updating input with new words.

2. Step 2: Identify correct iterative approach

   Feed the model the initial seed sequence, predict the next word, append it, then use the updated sequence to predict again describes feeding seed, predicting next word, appending it, and repeating, which is correct.

3. Final Answer:

   Feed the model the initial seed sequence, predict the next word, append it, then use the updated sequence to predict again -> Option A

4. Quick Check:

   Generate word-by-word with updated input = D [OK]

Hint: Generate text stepwise, updating input each time [OK]

Common Mistakes:

• Trying to generate all words at once
• Ignoring the need to update input sequence
• Selecting words randomly without model