Time scaling (compression and expansion) in Signal Processing - Deep Dive

Overview - Time scaling (compression and expansion)

What is it?

Time scaling is a process that changes the speed of a signal without altering its pitch. Compression makes the signal play faster, shortening its duration, while expansion slows it down, lengthening the duration. This technique is common in audio and speech processing to adjust playback speed. It helps us hear sounds faster or slower while keeping the original tone intact.

Why it matters

Without time scaling, changing the speed of audio or signals would also change their pitch, making voices sound unnatural or music distorted. Time scaling allows us to speed up or slow down signals for better understanding, analysis, or creative effects without losing the original character. This is important in areas like language learning, music production, and medical signal analysis.

Where it fits

Before learning time scaling, you should understand basic signal concepts like sampling and frequency. After mastering time scaling, you can explore advanced topics like pitch shifting, time-frequency analysis, and real-time audio effects.

Mental Model

Core Idea

Time scaling changes how fast or slow a signal plays without changing its pitch or tone.

Think of it like...

Imagine watching a movie where the actors move faster or slower, but their voices sound the same pitch, not like chipmunks or deep monsters.

Original Signal ──▶ [Time Scaling] ──▶ Compressed or Expanded Signal

Duration: Longer <───────────────|───────────────> Shorter
Pitch: Constant ─────────────────────────────────────────────

Build-Up - 7 Steps

1

Foundation

Understanding Signal Duration and Speed

Concept: Learn what signal duration and playback speed mean in simple terms.

A signal is a sequence of data points over time. Duration is how long the signal lasts. Speed is how fast we play the signal. Playing faster means shorter duration; playing slower means longer duration. Without special processing, changing speed also changes pitch.

Result

You can explain that playing a sound faster makes it shorter and higher-pitched, and playing slower makes it longer and lower-pitched.

Understanding duration and speed is key because time scaling aims to change speed without affecting pitch.

2

Foundation

What Happens When You Change Speed Directly

Concept: Explore the effect of simple speed changes on pitch and duration.

If you play a recorded sound faster by skipping samples, the sound becomes shorter and the pitch rises (like a chipmunk). Playing slower by repeating samples makes it longer and lowers pitch (like a slow monster voice). This is because speed and pitch are linked in raw playback.

Result

Direct speed change causes pitch distortion along with duration change.

Knowing this problem explains why time scaling needs special methods to keep pitch constant.

3

Intermediate

Basic Time Scaling Using Resampling

🤔Before reading on: Do you think resampling alone can change speed without affecting pitch? Commit to your answer.

Concept: Resampling changes the number of samples to speed up or slow down a signal but affects pitch.

Resampling means changing the sample rate of a signal. For example, playing a 44.1kHz signal at 22.05kHz doubles duration but lowers pitch. This method is simple but changes pitch along with speed.

Result

Resampling changes both speed and pitch, so it is not true time scaling.

Understanding resampling shows why more advanced techniques are needed for pitch preservation.

4

Intermediate

Time-Domain Techniques: Overlap-Add Method

🤔Before reading on: Do you think cutting and overlapping parts of a signal can change duration without pitch? Commit to your answer.

Concept: Overlap-Add (OLA) cuts the signal into small pieces and overlaps them to change duration while keeping pitch.

OLA splits the signal into short frames, then either skips or repeats frames to compress or expand time. Overlapping frames smoothly avoids gaps or jumps. This changes duration but keeps the frequency content stable, preserving pitch.

Result

You get a signal that plays faster or slower with nearly the same pitch.

Knowing OLA helps understand how time scaling can be done by rearranging signal pieces carefully.

5

Intermediate

Frequency-Domain Techniques: Phase Vocoder

🤔Before reading on: Can changing phase information in frequency domain help keep pitch constant during time scaling? Commit to your answer.

Concept: Phase vocoder uses frequency analysis to adjust timing while preserving pitch by controlling phase relationships.

The signal is transformed into frequency components using Fourier transform. The phase vocoder modifies the timing of these components by adjusting phases to stretch or compress time. This keeps the pitch stable because frequency content is preserved.

Result

Time-scaled signal with high quality pitch preservation, useful for music and speech.

Understanding phase vocoder reveals how frequency domain manipulation achieves better time scaling than time-domain methods.

6

Advanced

Challenges: Artifacts and Quality Trade-offs

🤔Before reading on: Do you think time scaling can be perfect without any sound artifacts? Commit to your answer.

Concept: Time scaling methods can introduce artifacts like echo, warbling, or unnatural sounds due to imperfect processing.

When frames overlap or phases shift incorrectly, artifacts appear. Compression can cause choppiness; expansion can cause reverberation. Balancing speed change and sound quality requires careful parameter tuning and sometimes hybrid methods.

Result

Real-world time scaling often involves trade-offs between speed change and audio quality.

Knowing artifact sources helps in choosing or designing better time scaling algorithms.

7

Expert

Real-Time Time Scaling and Computational Efficiency

🤔Before reading on: Is real-time time scaling computationally simple or does it require complex optimization? Commit to your answer.

Concept: Implementing time scaling in real-time systems requires efficient algorithms and low latency processing.

Real-time time scaling is used in live audio effects and communication. Algorithms must process data quickly with minimal delay. Techniques include optimized overlap-add, fast Fourier transforms, and hardware acceleration. Trade-offs between speed, quality, and resource use are critical.

Result

Real-time time scaling enables live speed adjustment without noticeable delay or quality loss.

Understanding real-time constraints reveals the complexity behind practical time scaling applications.

Under the Hood

Time scaling works by manipulating the signal's time axis while preserving its frequency content. In time-domain methods like Overlap-Add, the signal is split into frames that are overlapped and shifted to compress or expand time. Frequency-domain methods like the phase vocoder transform the signal into frequency components, adjust their phases to maintain pitch, and then reconstruct the time-scaled signal. These processes rely on careful control of phase and amplitude to avoid artifacts.

Why designed this way?

Early methods simply changed playback speed, causing pitch shifts. To keep pitch constant, researchers developed frame-based and frequency-domain techniques that separate time and frequency information. The design balances complexity, quality, and computational cost. Alternatives like simple resampling were rejected because they distort pitch, while more complex methods like granular synthesis trade off naturalness for flexibility.

Original Signal
    │
    ▼
[Frame Splitting] ──▶ [Overlap and Shift Frames] ──▶ [Reconstruction]
    │                                         ▲
    ▼                                         │
[Fourier Transform] ──▶ [Phase Adjustment] ──┘
    │
    ▼
Inverse Fourier Transform
    │
    ▼
Time-Scaled Signal

Myth Busters - 4 Common Misconceptions

Quick: Does speeding up a signal always raise its pitch? Commit to yes or no.

Common Belief: Speeding up a signal always raises its pitch and slowing down always lowers it.

Reality: With time scaling techniques, speed can change without affecting pitch.

Why it matters: Believing this limits understanding of how modern audio tools work and prevents using time scaling effectively.

Quick: Can resampling alone achieve time scaling without pitch change? Commit to yes or no.

Common Belief: Resampling can change speed without affecting pitch.

Reality: Resampling changes both speed and pitch simultaneously.

Why it matters: Misusing resampling leads to distorted audio and poor quality results.

Quick: Is time scaling always artifact-free? Commit to yes or no.

Common Belief: Time scaling can be done perfectly without any sound artifacts.

Reality: All practical time scaling methods introduce some artifacts or quality trade-offs.

Why it matters: Expecting perfect results causes frustration and misunderstanding of algorithm limitations.

Quick: Does time scaling only apply to audio signals? Commit to yes or no.

Common Belief: Time scaling is only useful for audio and music.

Reality: Time scaling applies to many signals, including speech, medical signals, and sensor data.

Why it matters: Limiting time scaling to audio misses its broader applications in science and engineering.

Expert Zone

1

Time scaling quality depends heavily on window size and overlap parameters, which affect artifact presence and computational load.

2

Phase vocoder methods can cause 'phasiness' artifacts, which experts mitigate using advanced phase-locking techniques.

3

Hybrid methods combining time-domain and frequency-domain approaches often yield better results than either alone.

When NOT to use

Time scaling is not suitable when exact original timing is critical, such as in synchronized multi-track recordings or certain medical diagnostics. Alternatives like pitch shifting or granular synthesis may be better depending on the goal.

Production Patterns

In production, time scaling is used in podcast editing to speed up speech without changing voice tone, in music remixing to match tempos, and in hearing aids to slow down speech for clarity. Real-time implementations use optimized overlap-add with hardware acceleration for low latency.

Connections

Fourier Transform

Time scaling in frequency domain builds directly on Fourier Transform concepts.

Understanding Fourier Transform helps grasp how signals can be manipulated in frequency to preserve pitch during time scaling.

Granular Synthesis

Granular synthesis is an alternative time scaling method using small sound grains.

Knowing granular synthesis provides perspective on different approaches to time scaling and their trade-offs.

Human Perception of Sound

Time scaling exploits how humans perceive pitch and timing separately.

Understanding auditory perception explains why pitch preservation during speed changes is important for natural sound.

Common Pitfalls

#1 Changing playback speed by simple resampling to speed up audio.

Wrong approach:

new_signal = resample(original_signal, new_rate=original_rate * 2) # doubles speed but pitch rises

Correct approach:

new_signal = time_scale(original_signal, scale=0.5) # compresses time without pitch change using time scaling algorithm

Root cause:

Confusing resampling with time scaling and not accounting for pitch change.

#2 Using too large frame size in Overlap-Add causing echo artifacts.

Wrong approach:

frame_size = 2048 # too large for speech signals
processed_signal = overlap_add(signal, frame_size)

Correct approach:

frame_size = 256 # smaller frame size reduces artifacts
processed_signal = overlap_add(signal, frame_size)

Root cause:

Not tuning frame size to signal characteristics leads to poor artifact control.

#3 Ignoring phase continuity in frequency domain time scaling causing warbling sound.

Wrong approach:

Modify magnitude spectrum only without adjusting phase during processing.

Correct approach:

Adjust phase spectrum carefully to maintain continuity during time scaling.

Root cause:

Misunderstanding importance of phase information in pitch preservation.

Key Takeaways

Time scaling changes the speed of a signal without changing its pitch, enabling natural-sounding speed adjustments.

Simple speed changes affect pitch, so specialized methods like Overlap-Add and phase vocoder are needed for true time scaling.

Time-domain and frequency-domain techniques each have strengths and weaknesses, often combined for best results.

Artifacts are common in time scaling, so parameter tuning and method choice are critical for quality.

Real-time time scaling requires efficient algorithms balancing speed, quality, and computational resources.

Practice

1. What happens to a signal when it is time-scaled with a factor a > 1?

easy

A. The signal is expanded and plays slower.

B. The signal amplitude increases.

C. The signal is compressed and plays faster.

D. The signal frequency decreases.

Solution

1. Step 1: Understand time scaling factor meaning

   A factor a > 1 means the time axis is divided by a number greater than 1, making the signal shorter in time.

2. Step 2: Effect on signal playback speed

   Shorter time means the signal plays faster, which is called compression.

3. Final Answer:

   The signal is compressed and plays faster. -> Option C

4. Quick Check:

   Compression (a > 1) = Faster playback [OK]

Hint: If a > 1, signal speeds up (compresses) [OK]

Common Mistakes:

• Confusing compression with expansion
• Thinking amplitude changes with time scaling
• Mixing frequency effects with time scaling

2. Which of the following Python code snippets correctly applies time scaling by a factor a to a signal array x using interpolation?

easy

A. import numpy as np
n = len(x)
t = np.arange(n)
t_new = t / a
x_new = x[t_new]

B. import numpy as np
n = len(x)
t = np.arange(n)
t_new = t * a
x_new = x[t_new]

C. import numpy as np
from scipy.interpolate import interp1d
n = len(x)
t = np.arange(n)
f = interp1d(t, x)
t_new = t * a
x_new = f(t_new)

D. import numpy as np
from scipy.interpolate import interp1d
n = len(x)
t = np.arange(n)
f = interp1d(t, x)
t_new = t / a
x_new = f(t_new)

Solution

1. Step 1: Understand time scaling formula

   Time scaling by factor a means new time points are t_new = t / a.

2. Step 2: Use interpolation for non-integer indices

   Since t_new may be fractional, interpolation function f is used to get x_new.

3. Final Answer:

   Code snippet A correctly applies time scaling with interpolation. -> Option D

4. Quick Check:

   t_new = t / a with interp1d = Correct scaling [OK]

Hint: Use t_new = t / a with interpolation function [OK]

Common Mistakes:

• Multiplying time by a instead of dividing
• Indexing array directly with fractional indices
• Forgetting to import interp1d

3. Given the code below, what is the length of the output signal x_scaled?

import numpy as np
from scipy.interpolate import interp1d
x = np.array([1, 2, 3, 4, 5])
a = 0.5
n = len(x)
t = np.arange(n)
f = interp1d(t, x, kind='linear', fill_value='extrapolate')
t_new = t / a
x_scaled = f(t_new)

medium

A. 5

B. 10

C. 3

D. 7

Solution

1. Step 1: Check length of t_new

   Variable t_new is computed as t / a, but t has length 5, so t_new also has length 5.

2. Step 2: Interpolation output length

   Interpolating at 5 points returns an array x_scaled of length 5.

3. Final Answer:

   5 -> Option A

4. Quick Check:

   Length of t_new = length of x_scaled = 5 [OK]

Hint: Output length matches length of t_new array [OK]

Common Mistakes:

• Assuming output length changes with scaling factor
• Confusing scaling factor with output size
• Ignoring that t_new length equals original length

4. Identify the error in the following code that attempts to expand a signal by a factor of 2:

import numpy as np
from scipy.interpolate import interp1d
x = np.array([10, 20, 30, 40])
a = 2
n = len(x)
t = np.arange(n)
f = interp1d(t, x)
t_new = t * a
x_expanded = f(t_new)

medium

A. Using integer array t instead of float array.

B. Using t * a instead of t / a for time scaling.

C. Using interp1d without sorting t.

D. Not specifying fill_value in interp1d causes error.

Solution

1. Step 1: Recall time scaling formula

   To expand (slow down) a signal by factor 2, new time points should be t_new = t / a, not t * a.

2. Step 2: Effect of incorrect scaling

   Using t * a compresses the signal instead of expanding it, causing wrong output.

3. Final Answer:

   Using t * a instead of t / a for time scaling. -> Option B

4. Quick Check:

   Expansion requires t_new = t / a [OK]

Hint: Use t_new = t / a for expansion, not t * a [OK]

Common Mistakes:

• Multiplying time by a instead of dividing
• Ignoring interpolation domain limits
• Not handling extrapolation in interp1d

5. You have a signal sampled at 1000 Hz with 1000 samples. You want to create a time-scaled version that plays twice as slow (expansion). Which of the following code snippets correctly computes the new signal x_expanded with proper interpolation and new sampling points?

hard

A. import numpy as np
from scipy.interpolate import interp1d
x = original_signal
fs = 1000
n = len(x)
t = np.arange(n) / fs
f = interp1d(t, x, kind='cubic', fill_value='extrapolate')
t_new = np.arange(0, t[-1]*2, 1/fs)
x_expanded = f(t_new)

B. import numpy as np
from scipy.interpolate import interp1d
x = original_signal
fs = 1000
n = len(x)
t = np.arange(n) / fs
f = interp1d(t, x, kind='linear')
t_new = np.arange(0, t[-1]/2, 1/fs)
x_expanded = f(t_new)

C. import numpy as np
from scipy.interpolate import interp1d
x = original_signal
fs = 1000
n = len(x)
t = np.arange(n) / fs
f = interp1d(t, x)
t_new = np.arange(0, t[-1]*2, 2/fs)
x_expanded = f(t_new)

D. import numpy as np
from scipy.interpolate import interp1d
x = original_signal
fs = 1000
n = len(x)
t = np.arange(n) / fs
f = interp1d(t, x, kind='cubic')
t_new = np.arange(0, t[-1]*0.5, 1/fs)
x_expanded = f(t_new)

Solution

1. Step 1: Understand expansion by factor 2

   Playing twice as slow means the signal duration doubles, so new time axis goes from 0 to t[-1]*2.

2. Step 2: Create new time points at original sampling rate

   New time points should be spaced by 1/fs to keep sampling rate consistent.

3. Step 3: Use interpolation with extrapolation

   Interpolation function with cubic method and fill_value='extrapolate' handles points beyond original range.

4. Final Answer:

   Code snippet A correctly computes the expanded signal. -> Option A

5. Quick Check:

   Expansion doubles duration with proper interpolation [OK]

Hint: Double time axis length, keep sampling interval, interpolate [OK]

Common Mistakes:

• Reducing time axis length instead of expanding
• Changing sampling interval incorrectly
• Not handling extrapolation in interpolation