Text-to-speech generation in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Text-to-speech generation

Problem:Create a model that converts text into natural-sounding speech audio.

Current Metrics:Training loss: 0.15, Validation loss: 0.45, Validation audio quality score: 3.2/5

Issue:The model overfits: training loss is low but validation loss is high, and generated speech sounds robotic and unnatural.

Your Task

Reduce overfitting to improve validation audio quality score from 3.2 to at least 4.0 while keeping training loss below 0.25.

You can only modify the model architecture and training hyperparameters.

Do not change the dataset or input preprocessing.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

Prompt Engineering / GenAI

import tensorflow as tf
from tensorflow.keras import layers, models

# Sample simplified TTS model architecture

def build_tts_model():
    inputs = layers.Input(shape=(None,))  # Input is sequence of text tokens
    x = layers.Embedding(input_dim=1000, output_dim=64)(inputs)
    x = layers.Bidirectional(layers.LSTM(128, return_sequences=True))(x)
    x = layers.BatchNormalization()(x)
    x = layers.Dropout(0.3)(x)
    x = layers.Bidirectional(layers.LSTM(64))(x)
    x = layers.BatchNormalization()(x)
    x = layers.Dropout(0.3)(x)
    outputs = layers.Dense(80, activation='linear')(x)  # Mel-spectrogram frame output
    model = models.Model(inputs, outputs)
    return model

model = build_tts_model()
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0005), loss='mse')

# Dummy data for demonstration (replace with real text token sequences and mel-spectrograms)
import numpy as np
X_train = np.random.randint(0, 1000, (500, 50))
y_train = np.random.rand(500, 80)

history = model.fit(X_train, y_train, epochs=30, batch_size=32, validation_split=0.3)

# After training, evaluate validation loss and listen to generated audio by converting predicted mel-spectrograms to audio using vocoder (not shown here)

Added dropout layers with rate 0.3 after LSTM layers to reduce overfitting.

Added batch normalization layers to stabilize and speed up training.

Reduced learning rate from 0.001 to 0.0005 for smoother convergence.

Increased validation split from 0.2 to 0.3 to better monitor validation performance.

Results Interpretation

Before: Training loss = 0.15, Validation loss = 0.45, Audio quality = 3.2/5

After: Training loss = 0.22, Validation loss = 0.30, Audio quality = 4.1/5

Adding dropout and batch normalization helped reduce overfitting, improving validation loss and audio quality. Lowering learning rate allowed the model to learn more smoothly, resulting in better generalization.

Bonus Experiment

Try using a different model architecture like a Transformer-based TTS model to further improve audio naturalness.

💡 Hint

Transformers can capture long-range dependencies in text better, which may improve speech prosody and clarity.

Practice

1. What is the main purpose of text-to-speech (TTS) technology?

easy

A. To summarize long documents automatically

B. To translate text from one language to another

C. To detect emotions in spoken language

D. To convert written text into spoken audio

Solution

1. Step 1: Understand the function of TTS

   Text-to-speech technology changes written words into sound that can be heard.

2. Step 2: Compare options with TTS purpose

   Only To convert written text into spoken audio describes converting text to speech, which matches TTS.

3. Final Answer:

   To convert written text into spoken audio -> Option D

4. Quick Check:

   TTS = convert text to speech [OK]

Hint: Remember TTS means text becomes speech [OK]

Common Mistakes:

• Confusing TTS with translation
• Thinking TTS summarizes text
• Mixing TTS with emotion detection

2. Which Python library is commonly used for simple text-to-speech conversion?

easy

A. Pandas

B. gTTS

C. Matplotlib

D. NumPy

Solution

1. Step 1: Identify libraries related to TTS

   gTTS is a Python library designed for text-to-speech conversion.

2. Step 2: Eliminate unrelated libraries

   NumPy, Matplotlib, and Pandas are for math, plotting, and data, not TTS.

3. Final Answer:

   gTTS -> Option B

4. Quick Check:

   gTTS = text-to-speech library [OK]

Hint: gTTS stands for Google Text-to-Speech [OK]

Common Mistakes:

• Choosing data or plotting libraries by mistake
• Confusing gTTS with general Python packages
• Assuming TTS needs complex libraries always

3. What will the following Python code output?

from gtts import gTTS
text = 'Hello world'
tts = gTTS(text)
tts.save('hello.mp3')
print('Audio saved')

medium

A. An audio file named 'hello.mp3' is created and 'Audio saved' is printed

B. The text 'Hello world' is printed on screen

C. A syntax error occurs due to missing language parameter

D. Nothing happens because gTTS requires internet connection

Solution

1. Step 1: Analyze the code steps

   The code imports gTTS, creates speech from 'Hello world', saves it as 'hello.mp3', then prints a message.

2. Step 2: Check for errors or missing parts

   gTTS defaults to English if no language is given, so no syntax error occurs. Internet is needed but code runs assuming connection.

3. Final Answer:

   An audio file named 'hello.mp3' is created and 'Audio saved' is printed -> Option A

4. Quick Check:

   Code saves audio and prints message [OK]

Hint: gTTS saves audio file and prints confirmation [OK]

Common Mistakes:

• Thinking language parameter is mandatory
• Assuming print outputs the text spoken
• Ignoring that gTTS needs internet but code runs

4. Identify the error in this text-to-speech code snippet:

from gtts import gTTS
tts = gTTS('Hello')
tts.save()

medium

A. Missing filename argument in save() method

B. gTTS requires language parameter in constructor

C. Text argument should be a list, not a string

D. gTTS cannot be imported directly

Solution

1. Step 1: Check gTTS usage

   gTTS constructor accepts text string; language is optional. So no error there.

2. Step 2: Check save() method

   save() requires a filename string argument to save the audio file. Missing argument causes error.

3. Final Answer:

   Missing filename argument in save() method -> Option A

4. Quick Check:

   save() needs filename [OK]

Hint: save() always needs a filename string [OK]

Common Mistakes:

• Assuming language is always required
• Thinking text must be a list
• Believing import statement is wrong

5. You want to create a text-to-speech system that can speak multiple languages based on user input. Which approach is best?

hard

A. Use gTTS without language parameter and rely on default English

B. Manually translate text first, then use gTTS with fixed language

C. Use gTTS with a dynamic language parameter set from user input

D. Use a single pre-recorded audio file for all languages

Solution

1. Step 1: Understand multilingual TTS needs

   The system must speak different languages based on user choice, so language must be flexible.

2. Step 2: Evaluate options for language flexibility

   Use gTTS with a dynamic language parameter set from user input sets language dynamically in gTTS, allowing correct speech for each language. Others fix language or use static audio, which won't adapt.

3. Final Answer:

   Use gTTS with a dynamic language parameter set from user input -> Option C

4. Quick Check:

   Dynamic language parameter enables multilingual TTS [OK]

Hint: Set language parameter dynamically for multilingual speech [OK]

Common Mistakes:

• Ignoring language parameter flexibility
• Assuming default English works for all
• Using static audio files for dynamic text