Spectrogram visualization in Signal Processing - Deep Dive

Overview - Spectrogram visualization

What is it?

A spectrogram is a visual way to show how the strength of different sound frequencies changes over time. It breaks a sound into small time pieces and shows which frequencies are loud or quiet in each piece. This creates a colorful picture where one axis is time, another is frequency, and colors show loudness. It helps us understand sounds beyond just listening.

Why it matters

Without spectrograms, we would only hear sounds but not see their hidden details. This makes it hard to analyze speech, music, or animal calls, or to detect problems in machines by their noise. Spectrograms let us spot patterns, changes, or problems quickly by looking, which is faster and clearer than just listening.

Where it fits

Before learning spectrograms, you should know basic sound concepts like frequency and amplitude, and how signals can be split into parts using Fourier transforms. After spectrograms, you can explore advanced audio analysis like speech recognition, music information retrieval, or machine fault diagnosis.

Mental Model

Core Idea

A spectrogram is like a heat map that shows how loud each sound frequency is at every moment in time.

Think of it like...

Imagine watching a rainbow-colored piano keyboard lighting up over time, where each key lights up brighter when its note is played louder. The horizontal direction is time, the vertical is the piano keys (frequencies), and the colors show how strong each note is.

Time →
Frequency ↑
┌─────────────────────────────┐
│ ░░░░░░░░░░░░░░░░░░░░░░░░░░ │
│ ░░▓▓▓▓▓░░░░░░░░░░░░░░░░░░░ │
│ ░░▓▓█████░░░░░░░░░░░░░░░░░ │
│ ░░░░███████░░░░░░░░░░░░░░░ │
│ ░░░░░░███████░░░░░░░░░░░░░ │
│ ░░░░░░░░███████░░░░░░░░░░░ │
│ ░░░░░░░░░░███████░░░░░░░░░ │
└─────────────────────────────┘
Color = loudness (dark = loud, light = quiet)

Build-Up - 7 Steps

1

Foundation

Understanding sound frequency basics

Concept: Sound is made of waves that vibrate at different speeds called frequencies.

Sound waves have high and low parts called frequencies. High frequency means a high-pitched sound like a whistle, and low frequency means a low-pitched sound like a drum. We can measure these frequencies in hertz (Hz), which counts how many waves pass per second.

Result

You can tell that sounds have different pitches because of their frequencies.

Understanding frequency is key because spectrograms show how these frequencies change over time.

2

Foundation

Time and frequency in signals

Concept: A sound changes over time, so we need to look at both when and what frequencies happen.

Sounds are not just one frequency but many mixed together. Over time, the mix changes. To study this, we split the sound into small time slices and look at the frequencies in each slice. This way, we see how the sound evolves.

Result

You realize that sound is dynamic and needs both time and frequency to describe it fully.

Knowing that sound changes over time sets the stage for why spectrograms show time on one axis and frequency on another.

3

Intermediate

Fourier transform for frequency analysis

🤔Before reading on: do you think a Fourier transform shows frequencies for the whole sound at once or over time slices? Commit to your answer.

Concept: Fourier transform breaks a signal into its frequency parts, but standard Fourier transform looks at the whole signal at once.

The Fourier transform is a math tool that tells us which frequencies are in a sound and how strong they are. However, it looks at the entire sound duration, so it doesn't show how frequencies change over time. This is like seeing all the ingredients in a soup but not knowing when each was added.

Result

You get a frequency breakdown but lose time information.

Understanding this limitation explains why we need a time-based version of Fourier transform for spectrograms.

4

Intermediate

Short-time Fourier transform (STFT)

🤔Before reading on: do you think cutting sound into small pieces before Fourier transform helps see frequency changes over time? Commit to your answer.

Concept: STFT splits sound into short time windows and applies Fourier transform to each, capturing frequency changes over time.

Instead of analyzing the whole sound at once, STFT cuts it into small overlapping pieces called windows. Each window is short enough to assume frequencies are stable, so Fourier transform on each window shows frequencies at that time. Putting these results side by side creates a spectrogram.

Result

You get a 2D map of frequency vs. time showing how sound changes.

Knowing STFT is the core method behind spectrograms helps you understand their time-frequency tradeoff.

5

Intermediate

Window size and resolution tradeoff

🤔Before reading on: do you think bigger windows give better time or frequency detail? Commit to your answer.

Concept: Window size affects how well we see time changes versus frequency details in a spectrogram.

A bigger window captures more sound, giving better frequency detail but worse time detail because it averages over longer time. A smaller window shows quick changes in time but blurs frequency details. This tradeoff is like choosing between a sharp photo of a moving object or a blurry one that shows more colors.

Result

You learn to balance window size based on what detail matters more.

Understanding this tradeoff is crucial for making useful spectrograms tailored to your analysis needs.

6

Advanced

Color mapping and interpretation

🤔Before reading on: do you think brighter colors always mean louder sounds in spectrograms? Commit to your answer.

Concept: Spectrogram colors represent loudness but depend on the color map and scaling used.

Spectrograms use colors or grayscale to show loudness of frequencies. Different color maps (like heat or grayscale) and scaling (linear or logarithmic) affect how we see loudness differences. Log scale often helps see quiet sounds better. Interpreting colors correctly is key to understanding the sound's structure.

Result

You can read spectrograms accurately and avoid misinterpretation.

Knowing how color maps work prevents wrong conclusions about sound intensity.

7

Expert

Advanced spectrograms and limitations

🤔Before reading on: do you think spectrograms can perfectly show all sound details? Commit to your answer.

Concept: Spectrograms have limits like time-frequency resolution and can be enhanced by methods like wavelets or reassigned spectrograms.

Spectrograms cannot show perfect detail due to the time-frequency tradeoff and windowing effects. Advanced methods like wavelet transforms or reassigned spectrograms improve resolution or clarity. Also, noise and overlapping sounds can make interpretation tricky. Experts choose methods based on the problem and understand these limits.

Result

You appreciate spectrogram strengths and weaknesses and know when to use advanced tools.

Recognizing spectrogram limits helps avoid overconfidence and guides better analysis choices.

Under the Hood

Internally, a spectrogram is created by sliding a window over the sound signal, multiplying the signal in that window by a shape function (window function), then applying Fourier transform to find frequency components. This process repeats for each window position, producing a matrix of frequency magnitudes over time. The window function reduces edge effects, and overlapping windows smooth transitions. The resulting matrix is then converted to colors for visualization.

Why designed this way?

Spectrograms were designed to solve the problem that standard Fourier transform loses time information. By using short windows, they balance time and frequency detail. The windowing and overlapping approach was chosen to reduce artifacts and provide a continuous view of sound evolution. Alternatives like wavelets exist but spectrograms remain popular for their simplicity and interpretability.

Sound signal ──▶ [Windowing] ──▶ [Fourier Transform] ──▶ Frequency data per window
       │                                         │
       └───────────── Sliding over time ─────────┘

Result: Matrix of frequency vs. time magnitudes

Visualization: Matrix values mapped to colors forming the spectrogram

Myth Busters - 4 Common Misconceptions

Quick: Does a spectrogram show exact frequencies present at every instant? Commit yes or no.

Common Belief: Spectrograms show the exact frequencies present at every moment in time.

Reality: Spectrograms show frequency content averaged over short time windows, so they cannot pinpoint exact frequencies at an instant.

Why it matters: Believing this leads to overestimating spectrogram precision and misinterpreting rapid sound changes.

Quick: Do brighter colors always mean louder sounds regardless of settings? Commit yes or no.

Common Belief: Brighter colors in a spectrogram always mean louder sounds.

Reality: Color brightness depends on the color map and scaling; sometimes darker colors can represent louder sounds depending on settings.

Why it matters: Misreading colors can cause wrong conclusions about sound intensity or presence.

Quick: Does increasing window size improve both time and frequency resolution? Commit yes or no.

Common Belief: Using a bigger window improves both time and frequency resolution in a spectrogram.

Reality: Increasing window size improves frequency resolution but worsens time resolution due to averaging over longer periods.

Why it matters: Ignoring this tradeoff can cause poor analysis where either time or frequency details are lost.

Quick: Can spectrograms perfectly separate overlapping sounds? Commit yes or no.

Common Belief: Spectrograms can perfectly separate overlapping sounds in a recording.

Reality: Spectrograms show combined frequency content, so overlapping sounds appear mixed and cannot be perfectly separated visually.

Why it matters: Expecting perfect separation leads to frustration and misuse of spectrograms for source separation tasks.

Expert Zone

1

The choice of window function (Hann, Hamming, Blackman) affects sidelobe leakage and spectral clarity, which experts tune for specific signals.

2

Logarithmic frequency scales (like Mel scale) better match human hearing and are often used in speech and music analysis instead of linear scales.

3

Overlap percentage between windows balances smoothness and computational cost; typical values are 50% to 75%, but experts adjust based on signal characteristics.

When NOT to use

Spectrograms are not ideal when extremely high time or frequency resolution is needed simultaneously, or for non-stationary signals with very rapid changes. Alternatives like wavelet transforms or reassigned spectrograms provide better resolution. For source separation, specialized algorithms like independent component analysis are preferred.

Production Patterns

In real-world systems, spectrograms are used for voice activity detection, machine fault diagnosis by sound, music genre classification, and bioacoustic monitoring. They are often combined with machine learning models that take spectrogram images as input for automated analysis.

Connections

Fourier transform

Spectrograms build on Fourier transform by applying it to short time windows instead of the whole signal.

Understanding Fourier transform helps grasp how spectrograms reveal frequency content over time.

Image processing

Spectrograms are 2D images representing sound data, allowing image analysis techniques to be applied.

Knowing image processing enables advanced spectrogram analysis like pattern recognition and feature extraction.

Human auditory perception

Spectrogram frequency and amplitude scales can be adjusted to match how humans hear sound.

Connecting spectrograms to hearing science improves design of audio analysis systems that align with human perception.

Common Pitfalls

#1 Using too large a window size for sounds with quick changes.

Wrong approach:

window_size = 2048 # Very large window for fast-changing sound
spectrogram = stft(signal, window_size)

Correct approach:

window_size = 256 # Smaller window to capture quick changes
spectrogram = stft(signal, window_size)

Root cause:

Misunderstanding the time-frequency tradeoff leads to poor time resolution and missed details.

#2 Interpreting spectrogram colors without knowing the color map scale.

Wrong approach:

plt.imshow(spectrogram, cmap='viridis') # Without checking if brighter means louder

Correct approach:

plt.imshow(spectrogram, cmap='viridis', norm=LogNorm()) # Use log scale for better loudness representation

Root cause:

Ignoring color map and scaling causes misreading of loudness levels.

#3 Assuming spectrogram shows exact instantaneous frequencies.

Wrong approach:

peak_freq = np.argmax(spectrogram[:, current_time]) # Treat as exact frequency at that moment

Correct approach:

peak_freq = np.argmax(spectrogram[:, current_time]) # Interpret as dominant frequency over window duration

Root cause:

Not realizing spectrogram averages frequencies over time windows leads to overprecision.

Key Takeaways

Spectrograms visualize how sound frequencies change over time by splitting sound into short windows and analyzing each.

There is a fundamental tradeoff between time and frequency resolution controlled by window size.

Colors in spectrograms represent loudness but depend on color maps and scaling, so interpretation requires care.

Spectrograms are powerful but have limits; advanced methods exist for better resolution or specific tasks.

Understanding spectrograms connects signal processing, human hearing, and image analysis for rich audio insights.