Text-to-speech generation in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Text-to-speech generation

Which metric matters for Text-to-speech generation and WHY

For text-to-speech (TTS), the main goal is to create natural, clear, and understandable speech from text. Metrics that measure how close the generated speech sounds to real human speech are important. These include Mean Opinion Score (MOS), which is a human rating of speech quality, and Word Error Rate (WER), which checks how well speech recognition can understand the generated audio. Lower WER means clearer speech. Also, Mel Cepstral Distortion (MCD) measures how close the sound features are to real speech. These metrics help us know if the TTS sounds natural and is easy to understand.

Confusion matrix or equivalent visualization

Unlike classification tasks, TTS does not use a confusion matrix. Instead, we use Word Error Rate (WER) calculated by comparing the text recognized from generated speech to the original text.

Original text: "Hello world"
Recognized text: "Hello word"

WER = (Substitutions + Deletions + Insertions) / Number of words in original
WER = 1 / 2 = 0.5 (50%)

Lower WER means better speech clarity.
    

Precision vs Recall tradeoff with concrete examples

In TTS, precision and recall are not typical metrics. Instead, there is a tradeoff between naturalness and intelligibility. For example:

• If the model focuses too much on sounding natural (like human voice), it might produce unclear words, lowering intelligibility.
• If the model focuses too much on clear pronunciation, the speech might sound robotic and less natural.

Balancing these is key. High naturalness with low intelligibility means listeners enjoy the voice but can't understand it well. High intelligibility with low naturalness means clear words but a boring or robotic voice.

What "good" vs "bad" metric values look like for Text-to-speech

Good values:

• MOS: Around 4.0 to 5.0 (on a scale of 1 to 5) means listeners find the speech natural and pleasant.
• WER: Close to 0% means the speech is very clear and easy to understand.
• MCD: Lower values (e.g., below 5 dB) mean the generated speech closely matches real speech acoustics.

Bad values:

• MOS: Below 3.0 means the speech sounds unnatural or robotic.
• WER: High values (e.g., above 20%) mean the speech is hard to understand.
• MCD: High values mean the speech sounds very different from real human speech.

Common pitfalls in Text-to-speech metrics

• Relying only on automatic metrics: Metrics like WER and MCD do not capture how natural or pleasant the speech sounds. Human listening tests (MOS) are essential.
• Ignoring context: Some words or phrases may be harder to pronounce. Evaluating only on easy sentences can give a false sense of quality.
• Overfitting to training voices: The model may sound great on voices it trained on but poorly on new voices or accents.
• Data leakage: Using test sentences seen during training can inflate metric scores.

Self-check question

Your TTS model has a Mean Opinion Score (MOS) of 4.5 but a Word Error Rate (WER) of 30%. Is this good for production? Why or why not?

Answer: This means listeners find the speech very natural (high MOS), but the speech is hard to understand (high WER). For production, this is not good because users may enjoy the voice but struggle to understand the words. You should improve clarity (reduce WER) while keeping naturalness.

Key Result

Text-to-speech quality is best judged by a balance of naturalness (MOS) and clarity (WER), with low WER and high MOS indicating good performance.