Time series analysis patterns in Pandas - Time & Space Complexity

Time Complexity: Time series analysis patterns

O(n)

Understanding Time Complexity

When working with time series data, it is important to know how the time to analyze patterns grows as the data gets bigger.

We want to understand how the steps to find patterns change when we have more time points.

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

df = pd.DataFrame({
    'date': pd.date_range(start='2023-01-01', periods=1000, freq='D'),
    'value': range(1000)
})
df.set_index('date', inplace=True)
rolling_mean = df['value'].rolling(window=7).mean()
seasonal_diff = df['value'] - df['value'].shift(365)

This code calculates a 7-day rolling average and a seasonal difference with a 365-day lag on a time series.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: The rolling calculation and the shift operation both process each data point in the series.
• How many times: Each operation goes through all n data points once.

How Execution Grows With Input

As the number of time points increases, the work to compute rolling means and seasonal differences grows in a straight line.

Input Size (n)	Approx. Operations
10	About 10 operations
100	About 100 operations
1000	About 1000 operations

Pattern observation: Doubling the data roughly doubles the work needed.

Final Time Complexity

Time Complexity: O(n)

This means the time to analyze patterns grows directly with the number of data points.

Common Mistake

[X] Wrong: "Rolling calculations take constant time no matter the data size."

[OK] Correct: Each new data point requires updating the rolling window, so the total work grows with data size.

Interview Connect

Understanding how time series operations scale helps you explain your approach clearly and shows you know how to handle growing data efficiently.

Self-Check

"What if we changed the rolling window size from 7 to 30? How would the time complexity change?"

Practice

1. What is the main purpose of using a rolling average in time series analysis with pandas?

easy

A. To convert time series data into categorical data

B. To remove all missing values from the data

C. To smooth out short-term fluctuations and highlight longer-term trends

D. To sort the data by date in ascending order

Solution

1. Step 1: Understand rolling average concept

   A rolling average calculates the mean of data points in a moving window, smoothing short-term noise.

2. Step 2: Identify its effect on time series data

   This smoothing helps reveal longer-term trends by reducing random fluctuations.

3. Final Answer:

   To smooth out short-term fluctuations and highlight longer-term trends -> Option C

4. Quick Check:

   Rolling average = smoothing trends [OK]

Hint: Rolling average smooths data to show trends clearly [OK]

Common Mistakes:

• Thinking rolling average removes missing values
• Confusing rolling average with sorting
• Assuming it changes data type

2. Which of the following is the correct pandas syntax to calculate a 7-day rolling mean on a DataFrame column df['sales']?

easy

A. df['sales'].rolling(window=7).mean()

B. df['sales'].rolling(7).sum()

C. df.rolling(window=7)['sales'].sum()

D. df['sales'].rolling(window=7).median()

Solution

1. Step 1: Recall rolling syntax in pandas

   The rolling method is called on the Series with window=7 to specify 7 days.

2. Step 2: Apply mean function to get rolling average

   Using .mean() after rolling computes the rolling mean correctly.

3. Final Answer:

   df['sales'].rolling(window=7).mean() -> Option A

4. Quick Check:

   rolling(window=7).mean() = correct syntax [OK]

Hint: Use rolling(window=7).mean() for 7-day rolling average [OK]

Common Mistakes:

• Using rolling(7) without window= keyword
• Using sum() instead of mean() for average
• Applying rolling on DataFrame instead of Series

3. Given the following code, what will be the output of print(diff_series)?

import pandas as pd
s = pd.Series([10, 15, 20, 18, 25])
diff_series = s.diff()

medium

A. [NaN, 5.0, 5.0, -2.0, 7.0]

B. [10, 15, 20, 18, 25]

C. [NaN, 10, 15, 20, 18]

D. [5, 5, -2, 7, NaN]

Solution

1. Step 1: Understand what diff() does on a Series

   The diff() method calculates the difference between each element and its previous element.

2. Step 2: Calculate differences step-by-step

   Differences: NaN (no previous), 15-10=5, 20-15=5, 18-20=-2, 25-18=7.

3. Final Answer:

   [NaN, 5.0, 5.0, -2.0, 7.0] -> Option A

4. Quick Check:

   diff() = current - previous [OK]

Hint: diff() shows change from previous value [OK]

Common Mistakes:

• Forgetting first value is NaN
• Confusing diff() with cumulative sum
• Misordering differences

4. The following code throws an error. What is the cause?

import pandas as pd
dates = pd.date_range('2023-01-01', periods=5)
df = pd.DataFrame({'value': [1, 2, 3, 4, 5]}, index=dates)
rolling_sum = df.rolling(3).sum()
print(rolling_sum['value'])

medium

A. The rolling window size must be specified with window=3

B. There is no error; the code runs correctly

C. The rolling method cannot be used on DataFrames

D. The DataFrame index must be reset before rolling

Solution

1. Step 1: Check rolling syntax on DataFrame

   Using df.rolling(3).sum() is valid; window size can be passed as positional argument.

2. Step 2: Verify if code runs without error

   Rolling sum on DataFrame with datetime index works fine; no reset needed.

3. Final Answer:

   There is no error; the code runs correctly -> Option B

4. Quick Check:

   rolling(3) on DataFrame = valid [OK]

Hint: rolling() accepts window as positional argument [OK]

Common Mistakes:

• Assuming window= is mandatory
• Thinking index must be reset
• Believing rolling only works on Series

5. You have daily sales data in a DataFrame df with a datetime index. You want to detect if sales on a day are unusually high compared to the previous week. Which pandas method combination is best to identify these spikes?

hard

A. Sort the DataFrame by sales and pick top 7 days

B. Use diff() to find daily changes and select days with positive differences

C. Apply autocorrelation to find repeated patterns

D. Calculate 7-day rolling mean and then find days where sales exceed mean by a threshold

Solution

1. Step 1: Understand the goal to find spikes compared to previous week

   We want to compare each day's sales to the average of the last 7 days.

2. Step 2: Use rolling mean to get weekly average and compare

   Calculating 7-day rolling mean smooths data; days exceeding this mean by threshold indicate spikes.

3. Final Answer:

   Calculate 7-day rolling mean and then find days where sales exceed mean by a threshold -> Option D

4. Quick Check:

   Rolling mean + threshold = spike detection [OK]

Hint: Compare daily sales to 7-day rolling mean for spikes [OK]

Common Mistakes:

• Using diff() only finds changes, not spikes
• Sorting ignores time context
• Autocorrelation finds patterns, not spikes