Text generation with RNN in TensorFlow

Introduction

Text generation with RNN helps computers write text by learning patterns from example sentences. It can create new sentences that sound like the original text.

To write simple stories or poems automatically.

To autocomplete sentences while typing messages.

To generate chatbot replies that sound natural.

To create new song lyrics based on existing ones.

To help with language learning by generating practice sentences.

Syntax

TensorFlow

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embedding_dim, batch_input_shape=[batch_size, None]),
    tf.keras.layers.SimpleRNN(rnn_units, return_sequences=True, stateful=True, recurrent_initializer='glorot_uniform'),
    tf.keras.layers.Dense(vocab_size)
])

The model uses an Embedding layer to turn words into numbers it can understand.

SimpleRNN layer processes sequences of words one by one, remembering past words.

Examples

This example creates a model for a vocabulary of 1000 words, embedding size 64, and 128 RNN units.

TensorFlow

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(1000, 64),
    tf.keras.layers.SimpleRNN(128, return_sequences=True),
    tf.keras.layers.Dense(1000)
])

This example uses LSTM instead of SimpleRNN for better memory, with a larger vocabulary and embedding.

TensorFlow

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(5000, 256),
    tf.keras.layers.LSTM(512, return_sequences=True),
    tf.keras.layers.Dense(5000)
])

Sample Model

This program trains a simple RNN on a tiny text sample to learn word sequences. Then it generates new words starting from a given phrase.

TensorFlow

import tensorflow as tf
import numpy as np

# Sample text data
text = "hello world hello tensorflow hello world"

# Create vocabulary
vocab = sorted(set(text.split()))
vocab_size = len(vocab)

# Map words to integers
word2idx = {u:i for i, u in enumerate(vocab)}
idx2word = np.array(vocab)

# Convert text to integers
text_as_int = np.array([word2idx[w] for w in text.split()])

# Prepare training sequences
seq_length = 3
examples_per_epoch = len(text_as_int) - seq_length

inputs = []
labels = []
for i in range(examples_per_epoch):
    inputs.append(text_as_int[i:i+seq_length])
    labels.append(text_as_int[i+1:i+seq_length+1])

inputs = np.array(inputs)
labels = np.array(labels)

# Build the model
embedding_dim = 8
rnn_units = 16

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embedding_dim, batch_input_shape=[None, seq_length]),
    tf.keras.layers.SimpleRNN(rnn_units, return_sequences=True),
    tf.keras.layers.Dense(vocab_size)
])

model.compile(optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True))

# Train the model
model.fit(inputs, labels, epochs=100, verbose=0)

# Generate text function
def generate_text(model, start_string, num_generate=5):
    input_eval = [word2idx[s] for s in start_string.split()]
    input_eval = tf.expand_dims(input_eval, 0)  # batch size 1

    text_generated = []

    for _ in range(num_generate):
        predictions = model(input_eval)
        predictions = tf.squeeze(predictions, 0)

        predicted_id = tf.random.categorical(predictions[-1:], num_samples=1)[-1,0].numpy()

        input_eval = tf.concat([input_eval[:,1:], [[predicted_id]]], axis=1)

        text_generated.append(idx2word[predicted_id])

    return ' '.join(text_generated)

# Generate new words
start = "hello world hello"
generated = generate_text(model, start)
print(f"Starting text: '{start}'")
print(f"Generated text: '{generated}'")

Important Notes

Training on very small text limits quality; bigger datasets improve results.

SimpleRNN is easy to understand but LSTM or GRU layers often give better text generation.

Setting return_sequences=True lets the model predict each word in the sequence.

Summary

RNNs can learn patterns in text to generate new sentences.

Embedding layers convert words into numbers the model can use.

Training requires sequences of words and their next words as labels.

Practice

1. What is the main purpose of using an RNN in text generation tasks?

easy

A. To sort words alphabetically

B. To count the number of words in a sentence

C. To learn patterns in sequences of words and predict the next word

D. To translate text into images

Solution

1. Step 1: Understand RNN function in text

   RNNs process sequences and learn how words follow each other.

2. Step 2: Identify the goal of text generation

   The goal is to predict the next word based on previous words.

3. Final Answer:

   To learn patterns in sequences of words and predict the next word -> Option C

4. Quick Check:

   RNN predicts next word = B [OK]

Hint: RNNs handle sequences and predict next items [OK]

Common Mistakes:

• Thinking RNNs sort or count words
• Confusing RNN with image processing
• Assuming RNNs translate text directly

2. Which of the following is the correct way to add an embedding layer in a TensorFlow RNN model for text generation?

easy

A. tf.keras.layers.MaxPooling1D(pool_size=2)

B. tf.keras.layers.Dense(units=embedding_dim)

C. tf.keras.layers.Conv2D(filters=embedding_dim, kernel_size=3)

D. tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim)

Solution

1. Step 1: Identify embedding layer syntax

   Embedding layers use input_dim and output_dim to map words to vectors.

2. Step 2: Match correct TensorFlow layer

   tf.keras.layers.Embedding is the correct layer for word embeddings.

3. Final Answer:

   tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim) -> Option D

4. Quick Check:

   Embedding layer uses input_dim and output_dim = A [OK]

Hint: Embedding layer uses input_dim and output_dim parameters [OK]

Common Mistakes:

• Using Dense instead of Embedding for word vectors
• Confusing Conv2D with text embedding
• Applying MaxPooling1D incorrectly for embeddings

3. Given the following code snippet, what will be the shape of the output from the RNN layer?

model = tf.keras.Sequential([
  tf.keras.layers.Embedding(input_dim=1000, output_dim=64),
  tf.keras.layers.SimpleRNN(128, return_sequences=False),
  tf.keras.layers.Dense(1000, activation='softmax')
])

input_shape = (32, 10)  # batch size 32, sequence length 10
output = model(tf.random.uniform(input_shape, maxval=1000, dtype=tf.int32))
print(output.shape)

medium

A. (32, 1000)

B. (32, 10, 128)

C. (10, 128)

D. (32, 128)

Solution

1. Step 1: Analyze RNN layer output shape

   SimpleRNN with return_sequences=False outputs (batch_size, units) = (32, 128).

2. Step 2: Check Dense layer output shape

   Dense layer with 1000 units outputs (32, 1000).

3. Final Answer:

   (32, 1000) -> Option A

4. Quick Check:

   Final output shape = (batch_size, vocab_size) = C [OK]

Hint: return_sequences=False means RNN outputs last step only [OK]

Common Mistakes:

• Confusing return_sequences=True output shape
• Ignoring batch size in output shape
• Mixing RNN units with Dense output units

4. Which of the following statements is CORRECT about this TensorFlow RNN model code for text generation:

model = tf.keras.Sequential([
  tf.keras.layers.Embedding(input_dim=5000, output_dim=64),
  tf.keras.layers.SimpleRNN(128, return_sequences=True),
  tf.keras.layers.Dense(5000, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

# Training data: input sequences of shape (batch_size, seq_length)
# Labels: next word indices of shape (batch_size, seq_length)

model.fit(train_data, train_labels, epochs=5)

medium

A. Loss function sparse_categorical_crossentropy is incorrect for this task

B. Using return_sequences=True but labels shape is (batch_size, seq_length) which is correct

C. Labels shape should be (batch_size,) not (batch_size, seq_length)

D. Embedding layer input_dim is too small

Solution

1. Step 1: Check RNN output and labels shape

   return_sequences=True outputs sequences matching label shape (batch_size, seq_length).

2. Step 2: Verify loss function suitability

   sparse_categorical_crossentropy is correct for integer labels in classification.

3. Final Answer:

   Using return_sequences=True but labels shape is (batch_size, seq_length) which is correct -> Option B

4. Quick Check:

   return_sequences=True matches label shape = A [OK]

Hint: return_sequences=True means labels must be sequences too [OK]

Common Mistakes:

• Assuming labels must be 1D when output is sequence
• Choosing wrong loss function for integer labels
• Changing embedding input_dim unnecessarily

5. You want to generate text using a trained RNN model. Which approach correctly generates new text step-by-step?

hard

A. Feed initial seed text, predict next word, append it, then feed updated text repeatedly

B. Feed entire training dataset at once to get all predictions

C. Use the model to predict all words in one call without updating input

D. Randomly select words without using the model predictions

Solution

1. Step 1: Understand text generation process

   Text generation uses seed text and predicts one word at a time.

2. Step 2: Describe iterative prediction

   After predicting a word, append it to input and predict next word repeatedly.

3. Final Answer:

   Feed initial seed text, predict next word, append it, then feed updated text repeatedly -> Option A

4. Quick Check:

   Stepwise prediction with updated input = D [OK]

Hint: Generate text word-by-word, updating input each time [OK]

Common Mistakes:

• Trying to predict all words at once without feedback
• Ignoring seed text and using random words
• Feeding entire dataset instead of seed for generation