Audio transcription (Whisper) in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - Audio transcription (Whisper)

This pipeline converts spoken audio into written text using the Whisper model. It listens to audio, processes it, and outputs the transcription.

Data Flow - 5 Stages

1Audio Input

1 audio file, 30 seconds, 16kHz sample rate→Raw audio waveform loaded→1 audio array, 480,000 samples

Audio of a person saying 'Hello, how are you?'

↓

2Preprocessing

1 audio array, 480,000 samples→Convert waveform to log-Mel spectrogram→1 spectrogram, 80 Mel bands x 3000 frames

Spectrogram showing energy patterns of speech sounds

↓

3Feature Encoding

1 spectrogram, 80 x 3000→Encode spectrogram into feature vectors→1 feature tensor, 3000 x 512

Encoded features representing audio content

↓

4Decoder (Text Generation)

1 feature tensor, 3000 x 512→Generate text tokens from features→Sequence of text tokens, length 15

Tokens representing words: ['Hello', ',', 'how', 'are', 'you', '?']

↓

5Postprocessing

Sequence of text tokens, length 15→Convert tokens to readable text→String: 'Hello, how are you?'

Final transcription text

Training Trace - Epoch by Epoch

Loss
2.5 |****
2.0 |*** 
1.5 |**  
1.0 |*   
0.5 |    
     +----
      1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	2.3	0.45	Model starts learning basic audio-text alignment
2	1.8	0.60	Loss decreases, accuracy improves as model learns speech patterns
3	1.4	0.72	Model better understands phonemes and word boundaries
4	1.1	0.80	Improved transcription quality, fewer errors
5	0.9	0.85	Model converges with good transcription accuracy

Prediction Trace - 5 Layers

Layer 1: Audio Input

Layer 2: Preprocessing

Layer 3: Feature Encoding

Layer 4: Decoder (Text Generation)

Layer 5: Postprocessing

Model Quiz - 3 Questions

What does the preprocessing stage do to the audio?

A Turns audio into text tokens directly

B Converts audio into a spectrogram showing sound energy over time

C Splits audio into separate words

D Removes noise from the audio

Key Insight

This visualization shows how Whisper transforms raw audio into text by first converting sound waves into visual patterns, then encoding these into features, and finally decoding them into words. Training improves the model's ability to match audio to text, reducing errors over time.