Audio transcription (Whisper) in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Audio transcription (Whisper)

Which metric matters for Audio Transcription (Whisper) and WHY

For audio transcription, the main goal is to convert spoken words into text accurately. The key metric is Word Error Rate (WER). WER measures how many words the model got wrong compared to the true transcript. It counts substitutions, deletions, and insertions of words. A lower WER means better transcription quality.

WER is important because it directly shows how close the transcription is to the real speech. Other metrics like Character Error Rate (CER) can also be used, especially for languages without clear word boundaries.

Confusion Matrix or Equivalent Visualization

Unlike classification tasks, audio transcription does not use a confusion matrix. Instead, errors are counted as:

True Transcript:  "I love machine learning"
Model Output:    "I love machine"

Errors:
- Deletion: missing "learning"
- Substitution: none
- Insertion: none

WER = (Substitutions + Deletions + Insertions) / Number of words in true transcript
WER = (0 + 1 + 0) / 4 = 0.25 (25%)
    

This shows 1 error out of 4 words, so 25% WER.

Precision vs Recall Tradeoff (Adapted for Transcription)

In transcription, precision and recall relate to how many words are correctly recognized versus missed or wrongly added.

• Precision: Of all words the model wrote, how many are correct? High precision means few extra or wrong words.
• Recall: Of all words spoken, how many did the model capture? High recall means few missed words.

For example, in a meeting transcript, missing important words (low recall) can cause misunderstanding. Adding wrong words (low precision) can confuse the meaning.

Balancing precision and recall is key. Whisper models aim to minimize overall errors (WER), which balances these aspects.

What "Good" vs "Bad" Metric Values Look Like

Good transcription: WER below 10% means the transcript is very close to the original speech. Most words are correct, and the text is easy to understand.

Bad transcription: WER above 30% means many words are wrong, missing, or extra. The transcript may be confusing or unusable.

Context matters: For noisy audio or multiple speakers, a WER around 15-20% might still be acceptable.

Common Metrics Pitfalls

• Ignoring context: Some errors may be minor and not affect meaning, but WER treats all errors equally.
• Data leakage: Testing on audio the model has seen before inflates performance.
• Overfitting: Model performs well on training accents or speakers but poorly on new ones.
• Ignoring language specifics: Some languages have complex word boundaries affecting WER calculation.

Self-Check Question

Your Whisper model has a 98% accuracy on a simple yes/no speech test but a 40% Word Error Rate on a long conversation. Is it good for real use? Why or why not?

Answer: No, it is not good for real use. The high accuracy on a simple test shows it can recognize very limited speech well, but the 40% WER on real conversations means many words are wrong or missing. This makes the transcript unreliable for understanding or decision-making.

Key Result

Word Error Rate (WER) is the key metric; lower WER means better transcription accuracy.

Practice

1. What is the main purpose of the Whisper model in audio transcription?

easy

A. Translate text from one language to another

B. Convert spoken words in audio files into written text

C. Generate music from text descriptions

D. Detect objects in images

Solution

1. Step 1: Understand Whisper's function

   Whisper is designed to listen to audio and write down what it hears as text.

2. Step 2: Compare options to Whisper's purpose

   Only Convert spoken words in audio files into written text matches this function; others describe unrelated tasks.

3. Final Answer:

   Convert spoken words in audio files into written text -> Option B

4. Quick Check:

   Whisper transcribes speech to text [OK]

Hint: Whisper turns speech into text, not images or translations [OK]

Common Mistakes:

• Confusing transcription with translation
• Thinking Whisper generates images or music
• Mixing audio transcription with image recognition

2. Which of the following is the correct way to call the Whisper model's transcription method in Python?

easy

A. model.audio_transcribe()

B. model.transcript(audio_file)

C. model.transcribe_audio(audio_file)

D. model.transcribe(audio_file)

Solution

1. Step 1: Recall the official Whisper method name

   The method to get text from audio is called transcribe().

2. Step 2: Match method call syntax

   model.transcribe(audio_file) uses model.transcribe(audio_file), which is correct syntax.

3. Final Answer:

   model.transcribe(audio_file) -> Option D

4. Quick Check:

   Use transcribe() method for transcription [OK]

Hint: Remember method name is exactly 'transcribe' with parentheses [OK]

Common Mistakes:

• Using incorrect method names like 'transcript' or 'transcribe_audio'
• Omitting parentheses when calling the method
• Confusing method with attribute access

3. Given the following Python code using Whisper, what will be the output type of result?

model = whisper.load_model('small')
audio_path = 'speech.mp3'
result = model.transcribe(audio_path)
print(type(result))

medium

A.

B.

C.

D.

Solution

1. Step 1: Understand the output of transcribe()

   The transcribe() method returns a dictionary containing keys like 'text' with the transcription.

2. Step 2: Identify the Python type of the output

   Since the output holds multiple pieces of information, it is a dict, not a string or list.

3. Final Answer:

   <class 'dict'> -> Option C

4. Quick Check:

   Whisper transcribe returns dict with transcription text [OK]

Hint: Whisper returns a dict with keys, not just a string [OK]

Common Mistakes:

• Assuming output is a plain string of text
• Thinking output is a list of words
• Confusing tuple with dictionary

4. You run this code but get an error:

model = whisper.load_model('medium')
result = model.transcribe()

What is the likely cause of the error?

medium

A. Missing audio file argument in transcribe() call

B. Model size 'medium' is not supported

C. transcribe() method does not exist

D. Audio file path is incorrect

Solution

1. Step 1: Check method call requirements

   The transcribe() method requires an audio file path argument to process.

2. Step 2: Identify missing argument

   The code calls transcribe() without any argument, causing an error.

3. Final Answer:

   Missing audio file argument in transcribe() call -> Option A

4. Quick Check:

   transcribe() needs audio file input [OK]

Hint: Always pass audio file path to transcribe() [OK]

Common Mistakes:

• Forgetting to provide audio file argument
• Assuming model size 'medium' is invalid
• Thinking transcribe() needs no arguments

5. You want to transcribe a long audio file quickly but can accept slightly less accuracy. Which Whisper model size should you choose?

hard

A. tiny

B. medium

C. large

D. small

Solution

1. Step 1: Understand model size trade-offs

   Smaller models like 'tiny' are fastest but less accurate; larger models are slower but more accurate.

2. Step 2: Choose model balancing speed and accuracy

   'tiny' model offers the fastest transcription speed with acceptable accuracy trade-off for long audio.

3. Final Answer:

   tiny -> Option A

4. Quick Check:

   Choose 'tiny' for fastest transcription with some accuracy loss [OK]

Hint: Pick 'tiny' for fastest transcription with some accuracy trade-off [OK]

Common Mistakes:

• Choosing 'small' expecting fastest speed
• Picking 'large' for speed
• Confusing 'medium' as fastest