Window functions (expanding, ewm) in Data Analysis Python - Step-by-Step Execution

Concept Flow - Window functions (expanding, ewm)

Start with data series

↓

Choose window type: Expanding or EWM

↓

For each new data point:

↓

Expanding: Calculate stats over all data so far

↓

EWM: Calculate weighted stats with decay

↓

Output updated series with window stats

↓

End

We start with a data series, pick a window type (expanding or exponentially weighted), then for each new data point, calculate statistics over the window and output the updated series.

Execution Sample

Data Analysis Python

import pandas as pd
s = pd.Series([1, 2, 3, 4, 5])
expanding_mean = s.expanding().mean()
ewm_mean = s.ewm(alpha=0.5).mean()
print(expanding_mean)
print(ewm_mean)

Calculate the expanding mean and exponentially weighted mean of a simple series.

Execution Table

Step	Data Point	Expanding Window Data	Expanding Mean	EWM Calculation	EWM Mean
1	1	[1]	1.0	1	1.0
2	2	[1, 2]	1.5	0.5*2 + 0.5*1=1.5	1.5
3	3	[1, 2, 3]	2.0	0.5*3 + 0.5*1.5=2.25	2.25
4	4	[1, 2, 3, 4]	2.5	0.5*4 + 0.5*2.25=3.125	3.125
5	5	[1, 2, 3, 4, 5]	3.0	0.5*5 + 0.5*3.125=4.0625	4.0625
6	End	-	-	-	-

💡 Reached end of data series, no more points to process.

Variable Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	Final
s	[1,2,3,4,5]	[1,2,3,4,5]	[1,2,3,4,5]	[1,2,3,4,5]	[1,2,3,4,5]	[1,2,3,4,5]	[1,2,3,4,5]
expanding_mean	empty	1.0	1.5	2.0	2.5	3.0	[1.0,1.5,2.0,2.5,3.0]
ewm_mean	empty	1.0	1.5	2.25	3.125	4.0625	[1.0,1.5,2.25,3.125,4.0625]

Key Moments - 3 Insights

Why does the expanding mean change with every new data point?

Because expanding mean includes all data points from the start up to the current one, so it recalculates the average over a growing window each step (see execution_table rows 1-5).

How does the exponentially weighted mean (EWM) differ from the expanding mean?

EWM gives more weight to recent points using a decay factor (alpha). It updates the mean by combining the new point with the previous EWM mean, shown in the EWM Calculation column.

Why does EWM mean not simply average all points like expanding mean?

Because EWM uses a weighted formula that discounts older points exponentially, so older data has less influence than recent data (see EWM Calculation and EWM Mean columns).

Visual Quiz - 3 Questions

Look at the execution_table at Step 3. What is the expanding mean value?

A 3.0

B 2.0

C 1.5

D 2.25

💡 Hint

Check the 'Expanding Mean' column at Step 3 in the execution_table.

At which step does the EWM mean first exceed 3.0?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Look at the 'EWM Mean' column in the execution_table and find when it passes 3.0.

If alpha in EWM is increased, how would the EWM mean values change?

A They would give more weight to recent points, making EWM mean react faster.

B They would give more weight to older points.

C They would become the same as expanding mean.

D They would not change.

💡 Hint

Recall that alpha controls the decay rate in EWM, higher alpha means more weight on recent data.

Concept Snapshot

Window functions process data in chunks over time.
Expanding windows grow with each new point, averaging all data so far.
Exponentially weighted windows (EWM) weight recent data more using alpha.
Use expanding() or ewm(alpha) in pandas for these.
EWM reacts faster to recent changes than expanding.
Both help analyze trends in time series data.

Full Transcript

We start with a data series and choose a window function: expanding or exponentially weighted mean (EWM). For each new data point, expanding mean calculates the average of all points from the start up to that point, so the window grows. EWM calculates a weighted average giving more importance to recent points using a decay factor alpha. The execution table shows step-by-step how the expanding mean and EWM mean update as new data points arrive. Variables track the series and the calculated means after each step. Key moments clarify why expanding mean changes with every point, how EWM differs by weighting recent data more, and why EWM does not simply average all points. The quiz tests understanding of values at specific steps and effects of changing alpha. The snapshot summarizes that expanding windows grow and average all data so far, while EWM weights recent data more, useful for analyzing trends in time series.

Practice

1. What does the expanding() window function do in data analysis?

easy

A. Calculates statistics using a fixed-size sliding window

B. Calculates statistics using only the last 3 data points

C. Calculates statistics by giving more weight to recent data points

D. Calculates statistics using all data points up to the current one

Solution

1. Step 1: Understand expanding() function

   The expanding() function includes all data points from the start up to the current point for calculations.

2. Step 2: Compare with other window types

   Unlike rolling windows or ewm, expanding uses all past data, not a fixed window or weighted recent data.

3. Final Answer:

   Calculates statistics using all data points up to the current one -> Option D

4. Quick Check:

   expanding() = all data so far [OK]

Hint: expanding() uses all data up to now, no fixed window [OK]

Common Mistakes:

• Confusing expanding() with rolling() fixed window
• Thinking ewm() is the same as expanding()
• Assuming expanding() only uses recent data

2. Which of the following is the correct syntax to calculate the exponentially weighted mean of a pandas Series data with a span of 3?

easy

A. data.ewm(span=3).mean()

B. data.expanding(span=3).mean()

C. data.rolling(window=3).ewm().mean()

D. data.ewm(window=3).mean()

Solution

1. Step 1: Recall ewm() syntax

   The ewm() function uses parameters like span to control decay; span=3 is valid.

2. Step 2: Check each option

   data.ewm(span=3).mean() uses correct syntax: data.ewm(span=3).mean(). Others misuse parameters or combine methods incorrectly.

3. Final Answer:

   data.ewm(span=3).mean() -> Option A

4. Quick Check:

   ewm(span=3) is correct syntax [OK]

Hint: Use ewm(span=3).mean() for weighted mean with span 3 [OK]

Common Mistakes:

• Using expanding() instead of ewm() for weighted mean
• Passing window parameter to ewm() which is invalid
• Chaining rolling() before ewm() incorrectly

3. Given the pandas Series data = pd.Series([10, 20, 30, 40]), what is the output of data.expanding().mean().tolist()?

medium

A. [10.0, 15.0, 20.0, 25.0]

B. [10.0, 20.0, 30.0, 40.0]

C. [10.0, 15.0, 25.0, 35.0]

D. [10.0, 20.0, 25.0, 30.0]

Solution

1. Step 1: Calculate expanding mean step-by-step

   Expanding mean at each point is average of all values up to that index:
   Index 0: 10 / 1 = 10.0
   Index 1: (10+20)/2 = 15.0
   Index 2: (10+20+30)/3 = 20.0
   Index 3: (10+20+30+40)/4 = 25.0

2. Step 2: Match calculated list with options

   The list is [10.0, 15.0, 20.0, 25.0], which matches [10.0, 15.0, 20.0, 25.0] exactly.

3. Final Answer:

   [10.0, 15.0, 20.0, 25.0] -> Option A

4. Quick Check:

   Expanding mean averages all so far [OK]

Hint: Expanding mean averages all data up to current index [OK]

Common Mistakes:

• Calculating rolling mean instead of expanding mean
• Using last value only instead of average
• Miscomputing cumulative sums

4. You run the code data.ewm(span=2).mean() but get an error. Which fix will solve the problem?

medium

A. Import pandas as pd before using data

B. Replace span with window parameter

C. Use data.ewm(span=2).mean() on a pandas Series, not a list

D. Call mean() before ewm()

Solution

1. Step 1: Identify cause of error

   The ewm() method is a pandas Series method. If data is a plain list, it causes an error.

2. Step 2: Correct usage context

   Convert data to a pandas Series before calling ewm(), e.g., pd.Series(data).ewm(span=2).mean().

3. Final Answer:

   Use data.ewm(span=2).mean() on a pandas Series, not a list -> Option C

4. Quick Check:

   ewm() needs pandas Series [OK]

Hint: ewm() works only on pandas Series, not plain lists [OK]

Common Mistakes:

• Trying to use ewm() on Python lists
• Using wrong parameter names like window in ewm()
• Calling mean() before ewm()

5. You have daily sales data with some missing days. You want to smooth the sales trend giving more weight to recent days but still include all past data. Which pandas window function and parameter combination is best?

hard

A. data.expanding(min_periods=5).mean() to average all data after 5 days

B. data.ewm(span=5, adjust=False).mean() to weight recent days more

C. data.rolling(window=5).mean() to use last 5 days only

D. data.expanding().mean() to average all data equally

Solution

1. Step 1: Understand smoothing with weights

   To give more weight to recent days, ewm() is appropriate because it applies exponential decay.

2. Step 2: Consider missing days and all past data

   ewm() includes all past data but weights recent points more; adjust=False gives a smoother result.

3. Step 3: Compare options

   data.ewm(span=5, adjust=False).mean() to weight recent days more uses ewm(span=5, adjust=False) which fits the requirement best. Others either use equal weights or fixed windows.

4. Final Answer:

   data.ewm(span=5, adjust=False).mean() -> Option B

5. Quick Check:

   ewm() weights recent data more [OK]

Hint: Use ewm(span=5, adjust=False) for weighted smoothing with all past data [OK]

Common Mistakes:

• Using expanding() which weights all data equally
• Using rolling() which ignores older data beyond window
• Not setting adjust=False for smoother ewm