Time series analysis patterns in Pandas - Step-by-Step Execution

Concept Flow - Time series analysis patterns

Load time series data

↓

Check data frequency

↓

Identify trend pattern

↓

Identify seasonality

↓

Detect noise/randomness

↓

Apply smoothing or decomposition

↓

Analyze residuals and patterns

↓

Make forecasts or insights

This flow shows how to analyze time series data step-by-step: loading data, checking frequency, spotting trend and seasonality, handling noise, and preparing for forecasting.

Execution Sample

Pandas

import pandas as pd
import matplotlib.pyplot as plt

data = pd.Series([10,12,15,14,13,16,18,20,19,21],
                 index=pd.date_range('2023-01-01', periods=10))
rolling_mean = data.rolling(window=3).mean()
rolling_mean.plot()
plt.show()

This code creates a simple time series, calculates a 3-day rolling average to smooth data, and plots the smoothed series.

Execution Table

Step	Action	Data/Variable	Result/Output
1	Create time series data	data	[10,12,15,14,13,16,18,20,19,21] with dates 2023-01-01 to 2023-01-10
2	Calculate rolling mean with window=3	rolling_mean	[NaN, NaN, 12.33, 13.67, 14.00, 14.33, 15.67, 18.00, 19.00, 20.00]
3	Plot rolling mean	plot	Line graph showing smoothed trend over dates
4	Exit	-	Rolling mean calculated and plotted

💡 Rolling mean calculation ends after processing all data points; first two are NaN due to window size.

Variable Tracker

Variable	Start	After Step 1	After Step 2	Final
data	empty	[10,12,15,14,13,16,18,20,19,21]	[10,12,15,14,13,16,18,20,19,21]	[10,12,15,14,13,16,18,20,19,21]
rolling_mean	empty	empty	[NaN, NaN, 12.33, 13.67, 14.00, 14.33, 15.67, 18.00, 19.00, 20.00]	[NaN, NaN, 12.33, 13.67, 14.00, 14.33, 15.67, 18.00, 19.00, 20.00]

Key Moments - 3 Insights

Why are the first two values of rolling_mean NaN?

Because the rolling window size is 3, the first two data points don't have enough previous values to calculate the average, so they are NaN as shown in execution_table step 2.

What does the rolling mean help us see in the time series?

It smooths out short-term fluctuations (noise) to reveal the underlying trend, as seen in the plotted line in execution_table step 3.

Why do we use a date range as the index for the series?

Using dates as the index helps pandas recognize the data as time series, enabling time-based operations like rolling windows and plotting with correct time labels.

Visual Quiz - 3 Questions

Look at the execution_table step 2, what is the rolling mean value at the 5th data point (index 4)?

A 14.00

B 13.67

C 12.33

D NaN

💡 Hint

Check the rolling_mean values in execution_table step 2, column 'Result/Output' at position 5.

At which step does the code produce the plot showing the smoothed trend?

A Step 1

B Step 3

C Step 2

D Step 4

💡 Hint

Refer to execution_table 'Action' column to find when plotting happens.

If we change the rolling window from 3 to 5, how will the rolling_mean values change?

A No NaN values, same smoothing

B Fewer NaN values, more noise

C More NaN values at the start, smoother trend

D Rolling mean will be the same

💡 Hint

Think about how window size affects the number of initial NaNs and smoothing effect.

Concept Snapshot

Time series analysis patterns:
- Use pandas Series with datetime index
- Identify trend by smoothing (rolling mean)
- Rolling window size controls smoothness and NaNs
- Plot to visualize patterns
- Helps separate trend, seasonality, noise

Full Transcript

This visual execution shows how to analyze time series data using pandas. We start by creating a time series with dates as index. Then, we calculate a rolling mean with a window of 3 to smooth the data, which helps reveal the trend by reducing noise. The first two rolling mean values are NaN because there are not enough points to average. Finally, we plot the smoothed series to see the trend visually. This process is a basic pattern in time series analysis to understand data behavior over time.

Practice

1. What is the main purpose of using a rolling average in time series analysis with pandas?

easy

A. To convert time series data into categorical data

B. To remove all missing values from the data

C. To smooth out short-term fluctuations and highlight longer-term trends

D. To sort the data by date in ascending order

Solution

1. Step 1: Understand rolling average concept

   A rolling average calculates the mean of data points in a moving window, smoothing short-term noise.

2. Step 2: Identify its effect on time series data

   This smoothing helps reveal longer-term trends by reducing random fluctuations.

3. Final Answer:

   To smooth out short-term fluctuations and highlight longer-term trends -> Option C

4. Quick Check:

   Rolling average = smoothing trends [OK]

Hint: Rolling average smooths data to show trends clearly [OK]

Common Mistakes:

• Thinking rolling average removes missing values
• Confusing rolling average with sorting
• Assuming it changes data type

2. Which of the following is the correct pandas syntax to calculate a 7-day rolling mean on a DataFrame column df['sales']?

easy

A. df['sales'].rolling(window=7).mean()

B. df['sales'].rolling(7).sum()

C. df.rolling(window=7)['sales'].sum()

D. df['sales'].rolling(window=7).median()

Solution

1. Step 1: Recall rolling syntax in pandas

   The rolling method is called on the Series with window=7 to specify 7 days.

2. Step 2: Apply mean function to get rolling average

   Using .mean() after rolling computes the rolling mean correctly.

3. Final Answer:

   df['sales'].rolling(window=7).mean() -> Option A

4. Quick Check:

   rolling(window=7).mean() = correct syntax [OK]

Hint: Use rolling(window=7).mean() for 7-day rolling average [OK]

Common Mistakes:

• Using rolling(7) without window= keyword
• Using sum() instead of mean() for average
• Applying rolling on DataFrame instead of Series

3. Given the following code, what will be the output of print(diff_series)?

import pandas as pd
s = pd.Series([10, 15, 20, 18, 25])
diff_series = s.diff()

medium

A. [NaN, 5.0, 5.0, -2.0, 7.0]

B. [10, 15, 20, 18, 25]

C. [NaN, 10, 15, 20, 18]

D. [5, 5, -2, 7, NaN]

Solution

1. Step 1: Understand what diff() does on a Series

   The diff() method calculates the difference between each element and its previous element.

2. Step 2: Calculate differences step-by-step

   Differences: NaN (no previous), 15-10=5, 20-15=5, 18-20=-2, 25-18=7.

3. Final Answer:

   [NaN, 5.0, 5.0, -2.0, 7.0] -> Option A

4. Quick Check:

   diff() = current - previous [OK]

Hint: diff() shows change from previous value [OK]

Common Mistakes:

• Forgetting first value is NaN
• Confusing diff() with cumulative sum
• Misordering differences

4. The following code throws an error. What is the cause?

import pandas as pd
dates = pd.date_range('2023-01-01', periods=5)
df = pd.DataFrame({'value': [1, 2, 3, 4, 5]}, index=dates)
rolling_sum = df.rolling(3).sum()
print(rolling_sum['value'])

medium

A. The rolling window size must be specified with window=3

B. There is no error; the code runs correctly

C. The rolling method cannot be used on DataFrames

D. The DataFrame index must be reset before rolling

Solution

1. Step 1: Check rolling syntax on DataFrame

   Using df.rolling(3).sum() is valid; window size can be passed as positional argument.

2. Step 2: Verify if code runs without error

   Rolling sum on DataFrame with datetime index works fine; no reset needed.

3. Final Answer:

   There is no error; the code runs correctly -> Option B

4. Quick Check:

   rolling(3) on DataFrame = valid [OK]

Hint: rolling() accepts window as positional argument [OK]

Common Mistakes:

• Assuming window= is mandatory
• Thinking index must be reset
• Believing rolling only works on Series

5. You have daily sales data in a DataFrame df with a datetime index. You want to detect if sales on a day are unusually high compared to the previous week. Which pandas method combination is best to identify these spikes?

hard

A. Sort the DataFrame by sales and pick top 7 days

B. Use diff() to find daily changes and select days with positive differences

C. Apply autocorrelation to find repeated patterns

D. Calculate 7-day rolling mean and then find days where sales exceed mean by a threshold

Solution

1. Step 1: Understand the goal to find spikes compared to previous week

   We want to compare each day's sales to the average of the last 7 days.

2. Step 2: Use rolling mean to get weekly average and compare

   Calculating 7-day rolling mean smooths data; days exceeding this mean by threshold indicate spikes.

3. Final Answer:

   Calculate 7-day rolling mean and then find days where sales exceed mean by a threshold -> Option D

4. Quick Check:

   Rolling mean + threshold = spike detection [OK]

Hint: Compare daily sales to 7-day rolling mean for spikes [OK]

Common Mistakes:

• Using diff() only finds changes, not spikes
• Sorting ignores time context
• Autocorrelation finds patterns, not spikes