Time series analysis patterns in Pandas

Introduction

Time series analysis helps us understand data that changes over time. It shows patterns like trends and cycles to make better decisions.

Tracking daily sales to see if business is growing

Monitoring temperature changes over months

Checking website visitors hour by hour

Analyzing stock prices to find trends

Observing heart rate data during exercise

Syntax

Pandas

import pandas as pd

# Load time series data with a datetime index
series = pd.Series(data, index=pd.to_datetime(dates))

# Common pattern checks
series.plot()  # Visualize data
series.rolling(window=3).mean()  # Moving average to smooth data
series.diff()  # Differences to find changes
series.autocorr(lag=1)  # Correlation with lagged data

Time series data must have a datetime index for proper analysis.

Rolling windows help smooth short-term fluctuations to see trends.

Examples

This example creates a simple daily time series and plots it to see the pattern.

Pandas

import pandas as pd
import matplotlib.pyplot as plt

dates = pd.date_range('2024-01-01', periods=6)
data = [10, 12, 15, 14, 13, 16]
series = pd.Series(data, index=dates)

# Plot the time series
title = 'Daily values'
series.plot(title=title)
plt.show()

Calculates the 3-day moving average to smooth the data and highlight trends.

Pandas

rolling_avg = series.rolling(window=3).mean()
print(rolling_avg)

Finds the difference between each day and the previous day to see changes.

Pandas

diffs = series.diff()
print(diffs)

Measures how much today's value relates to yesterday's value.

Pandas

autocorr_value = series.autocorr(lag=1)
print(f'Autocorrelation with lag 1: {autocorr_value}')

Sample Program

This program creates a time series, plots it, calculates a moving average to smooth the data, finds daily changes, and measures autocorrelation to see how values relate over time.

Pandas

import pandas as pd
import matplotlib.pyplot as plt

# Create sample time series data
dates = pd.date_range('2024-01-01', periods=10)
data = [5, 7, 6, 8, 10, 9, 11, 13, 12, 14]
series = pd.Series(data, index=dates)

# Plot original data
series.plot(title='Original Time Series')
plt.show()

# Calculate 3-day moving average
moving_avg = series.rolling(window=3).mean()
print('3-day Moving Average:')
print(moving_avg)

# Calculate daily differences
diffs = series.diff()
print('\nDaily Differences:')
print(diffs)

# Calculate autocorrelation with lag 1
autocorr = series.autocorr(lag=1)
print(f'\nAutocorrelation (lag=1): {autocorr:.2f}')

Important Notes

Missing values can appear in rolling averages at the start because there is not enough data to fill the window.

Autocorrelation helps detect repeating patterns or trends in time series.

Plotting is a great first step to visually understand your time series data.

Summary

Time series patterns show how data changes over time.

Use rolling averages to smooth data and see trends.

Differences and autocorrelation reveal changes and relationships in time.

Practice

1. What is the main purpose of using a rolling average in time series analysis with pandas?

easy

A. To convert time series data into categorical data

B. To remove all missing values from the data

C. To smooth out short-term fluctuations and highlight longer-term trends

D. To sort the data by date in ascending order

Solution

1. Step 1: Understand rolling average concept

   A rolling average calculates the mean of data points in a moving window, smoothing short-term noise.

2. Step 2: Identify its effect on time series data

   This smoothing helps reveal longer-term trends by reducing random fluctuations.

3. Final Answer:

   To smooth out short-term fluctuations and highlight longer-term trends -> Option C

4. Quick Check:

   Rolling average = smoothing trends [OK]

Hint: Rolling average smooths data to show trends clearly [OK]

Common Mistakes:

• Thinking rolling average removes missing values
• Confusing rolling average with sorting
• Assuming it changes data type

2. Which of the following is the correct pandas syntax to calculate a 7-day rolling mean on a DataFrame column df['sales']?

easy

A. df['sales'].rolling(window=7).mean()

B. df['sales'].rolling(7).sum()

C. df.rolling(window=7)['sales'].sum()

D. df['sales'].rolling(window=7).median()

Solution

1. Step 1: Recall rolling syntax in pandas

   The rolling method is called on the Series with window=7 to specify 7 days.

2. Step 2: Apply mean function to get rolling average

   Using .mean() after rolling computes the rolling mean correctly.

3. Final Answer:

   df['sales'].rolling(window=7).mean() -> Option A

4. Quick Check:

   rolling(window=7).mean() = correct syntax [OK]

Hint: Use rolling(window=7).mean() for 7-day rolling average [OK]

Common Mistakes:

• Using rolling(7) without window= keyword
• Using sum() instead of mean() for average
• Applying rolling on DataFrame instead of Series

3. Given the following code, what will be the output of print(diff_series)?

import pandas as pd
s = pd.Series([10, 15, 20, 18, 25])
diff_series = s.diff()

medium

A. [NaN, 5.0, 5.0, -2.0, 7.0]

B. [10, 15, 20, 18, 25]

C. [NaN, 10, 15, 20, 18]

D. [5, 5, -2, 7, NaN]

Solution

1. Step 1: Understand what diff() does on a Series

   The diff() method calculates the difference between each element and its previous element.

2. Step 2: Calculate differences step-by-step

   Differences: NaN (no previous), 15-10=5, 20-15=5, 18-20=-2, 25-18=7.

3. Final Answer:

   [NaN, 5.0, 5.0, -2.0, 7.0] -> Option A

4. Quick Check:

   diff() = current - previous [OK]

Hint: diff() shows change from previous value [OK]

Common Mistakes:

• Forgetting first value is NaN
• Confusing diff() with cumulative sum
• Misordering differences

4. The following code throws an error. What is the cause?

import pandas as pd
dates = pd.date_range('2023-01-01', periods=5)
df = pd.DataFrame({'value': [1, 2, 3, 4, 5]}, index=dates)
rolling_sum = df.rolling(3).sum()
print(rolling_sum['value'])

medium

A. The rolling window size must be specified with window=3

B. There is no error; the code runs correctly

C. The rolling method cannot be used on DataFrames

D. The DataFrame index must be reset before rolling

Solution

1. Step 1: Check rolling syntax on DataFrame

   Using df.rolling(3).sum() is valid; window size can be passed as positional argument.

2. Step 2: Verify if code runs without error

   Rolling sum on DataFrame with datetime index works fine; no reset needed.

3. Final Answer:

   There is no error; the code runs correctly -> Option B

4. Quick Check:

   rolling(3) on DataFrame = valid [OK]

Hint: rolling() accepts window as positional argument [OK]

Common Mistakes:

• Assuming window= is mandatory
• Thinking index must be reset
• Believing rolling only works on Series

5. You have daily sales data in a DataFrame df with a datetime index. You want to detect if sales on a day are unusually high compared to the previous week. Which pandas method combination is best to identify these spikes?

hard

A. Sort the DataFrame by sales and pick top 7 days

B. Use diff() to find daily changes and select days with positive differences

C. Apply autocorrelation to find repeated patterns

D. Calculate 7-day rolling mean and then find days where sales exceed mean by a threshold

Solution

1. Step 1: Understand the goal to find spikes compared to previous week

   We want to compare each day's sales to the average of the last 7 days.

2. Step 2: Use rolling mean to get weekly average and compare

   Calculating 7-day rolling mean smooths data; days exceeding this mean by threshold indicate spikes.

3. Final Answer:

   Calculate 7-day rolling mean and then find days where sales exceed mean by a threshold -> Option D

4. Quick Check:

   Rolling mean + threshold = spike detection [OK]

Hint: Compare daily sales to 7-day rolling mean for spikes [OK]

Common Mistakes:

• Using diff() only finds changes, not spikes
• Sorting ignores time context
• Autocorrelation finds patterns, not spikes