RNN-based text generation in NLP

Introduction

RNN-based text generation helps computers write text by learning patterns from example sentences. It can create new sentences that sound like the original text.

When you want a computer to write stories or poems similar to a style you provide.

When you need to autocomplete sentences or suggest text in chat apps.

When generating code snippets based on previous code examples.

When creating chatbots that respond with natural language.

When experimenting with creative writing using AI.

Syntax

NLP

model = Sequential()
model.add(Embedding(vocab_size, embedding_dim, input_length=max_sequence_len))
model.add(SimpleRNN(units))
model.add(Dense(vocab_size, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Use Embedding layer to convert words into numbers the model can understand.

SimpleRNN layer processes the sequence of words step-by-step to learn patterns.

Examples

This example creates an RNN model for a vocabulary of 1000 words and sequences of length 10.

NLP

model = Sequential()
model.add(Embedding(1000, 64, input_length=10))
model.add(SimpleRNN(128))
model.add(Dense(1000, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Here, the model handles longer sequences (20 words) and a bigger vocabulary (5000 words).

NLP

model = Sequential()
model.add(Embedding(5000, 100, input_length=20))
model.add(SimpleRNN(256))
model.add(Dense(5000, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Sample Model

This program trains a small RNN model on a tiny text sample. It learns to predict the next word given two words. Then it generates three new words starting from 'hello world'.

NLP

import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, SimpleRNN, Dense
from tensorflow.keras.utils import to_categorical

# Sample text data
text = "hello world hello machine learning"

# Create a simple word index
words = sorted(set(text.split()))
word_to_index = {w: i for i, w in enumerate(words)}
index_to_word = {i: w for w, i in word_to_index.items()}

# Prepare sequences (input and target)
sequence_length = 2
sequences = []
tokens = text.split()
for i in range(len(tokens) - sequence_length):
    seq = tokens[i:i+sequence_length+1]
    sequences.append([word_to_index[w] for w in seq])

sequences = np.array(sequences)
X, y = sequences[:, :-1], sequences[:, -1]
y = to_categorical(y, num_classes=len(words))

# Build the model
model = Sequential()
model.add(Embedding(input_dim=len(words), output_dim=10, input_length=sequence_length))
model.add(SimpleRNN(20))
model.add(Dense(len(words), activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# Train the model
model.fit(X, y, epochs=100, verbose=0)

# Generate text function
def generate_text(seed_text, next_words=3):
    result = seed_text.split()
    for _ in range(next_words):
        encoded = [word_to_index[w] for w in result[-sequence_length:]]
        encoded = np.array(encoded).reshape(1, sequence_length)
        pred = model.predict(encoded, verbose=0)
        next_index = np.argmax(pred)
        next_word = index_to_word[next_index]
        result.append(next_word)
    return ' '.join(result)

# Generate new text
output_text = generate_text('hello world')
print(output_text)

Important Notes

Training on very small text limits the quality of generated text.

Increasing epochs or model size can improve results but needs more data.

SimpleRNN is easy to understand but LSTM or GRU layers often work better for text.

Summary

RNNs read text word by word to learn patterns for generating new text.

Embedding layers turn words into numbers the model can work with.

Training needs input sequences and the next word as the target.