Experiment - Text-to-speech generation

Problem:Create a model that converts text into natural-sounding speech audio.

Current Metrics:Training loss: 0.15, Validation loss: 0.45, Validation audio quality score: 3.2/5

Issue:The model overfits: training loss is low but validation loss is high, and generated speech sounds robotic and unnatural.

Your Task

Reduce overfitting to improve validation audio quality score from 3.2 to at least 4.0 while keeping training loss below 0.25.

You can only modify the model architecture and training hyperparameters.

Do not change the dataset or input preprocessing.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

Prompt Engineering / GenAI

import tensorflow as tf
from tensorflow.keras import layers, models

# Sample simplified TTS model architecture

def build_tts_model():
    inputs = layers.Input(shape=(None,))  # Input is sequence of text tokens
    x = layers.Embedding(input_dim=1000, output_dim=64)(inputs)
    x = layers.Bidirectional(layers.LSTM(128, return_sequences=True))(x)
    x = layers.BatchNormalization()(x)
    x = layers.Dropout(0.3)(x)
    x = layers.Bidirectional(layers.LSTM(64))(x)
    x = layers.BatchNormalization()(x)
    x = layers.Dropout(0.3)(x)
    outputs = layers.Dense(80, activation='linear')(x)  # Mel-spectrogram frame output
    model = models.Model(inputs, outputs)
    return model

model = build_tts_model()
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0005), loss='mse')

# Dummy data for demonstration (replace with real text token sequences and mel-spectrograms)
import numpy as np
X_train = np.random.randint(0, 1000, (500, 50))
y_train = np.random.rand(500, 80)

history = model.fit(X_train, y_train, epochs=30, batch_size=32, validation_split=0.3)

# After training, evaluate validation loss and listen to generated audio by converting predicted mel-spectrograms to audio using vocoder (not shown here)

Added dropout layers with rate 0.3 after LSTM layers to reduce overfitting.

Added batch normalization layers to stabilize and speed up training.

Reduced learning rate from 0.001 to 0.0005 for smoother convergence.

Increased validation split from 0.2 to 0.3 to better monitor validation performance.

Results Interpretation

Before: Training loss = 0.15, Validation loss = 0.45, Audio quality = 3.2/5

After: Training loss = 0.22, Validation loss = 0.30, Audio quality = 4.1/5

Adding dropout and batch normalization helped reduce overfitting, improving validation loss and audio quality. Lowering learning rate allowed the model to learn more smoothly, resulting in better generalization.

Bonus Experiment

Try using a different model architecture like a Transformer-based TTS model to further improve audio naturalness.

💡 Hint

Transformers can capture long-range dependencies in text better, which may improve speech prosody and clarity.