Why time-based analysis reveals trends in Data Analysis Python - Performance Analysis

Time Complexity: Why time-based analysis reveals trends

O(n * k)

Understanding Time Complexity

When we analyze data over time, we want to see how patterns change as more time passes.

We ask: How does the work needed to find trends grow when we have more time points?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

def calculate_moving_average(data, window_size):
    moving_averages = []
    for i in range(len(data) - window_size + 1):
        window = data[i : i + window_size]
        moving_averages.append(sum(window) / window_size)
    return moving_averages

# Example usage:
data = [10, 20, 30, 40, 50, 60, 70]
result = calculate_moving_average(data, 3)

This code calculates a moving average over a time series to reveal trends.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Looping through the data to calculate averages for each window.
• How many times: The loop runs once for each possible window in the data, about n times where n is data length.

How Execution Grows With Input

As the data size grows, the number of averages to calculate grows roughly the same.

Input Size (n)	Approx. Operations
10	About 8 calculations
100	About 98 calculations
1000	About 998 calculations

Pattern observation: The work grows roughly in a straight line as data size increases.

Final Time Complexity

Time Complexity: O(n * k)

This means the time to find trends grows directly with the amount of data points and the window size.

Common Mistake

[X] Wrong: "Calculating moving averages takes the same time no matter how much data there is."

[OK] Correct: More data means more windows to average, so the work grows with data size.

Interview Connect

Understanding how time-based data grows helps you explain how your analysis scales with more data.

Self-Check

"What if we used a cumulative sum to calculate moving averages instead? How would the time complexity change?"

Practice

1. Why is time-based analysis important in data science?

easy

A. It removes all missing data automatically.

B. It helps us see how data changes over time to find trends.

C. It ignores the order of data points.

D. It only works with numbers, not dates.

Solution

1. Step 1: Understand the role of time in data

   Time-based analysis focuses on data points collected over time to observe changes.

2. Step 2: Identify the benefit of time-based analysis

   By analyzing data over time, we can detect trends and patterns that are not visible in static data.

3. Final Answer:

   It helps us see how data changes over time to find trends. -> Option B

4. Quick Check:

   Time-based analysis reveals trends = D [OK]

Hint: Time shows how data evolves; trends need time order [OK]

Common Mistakes:

• Thinking time-based analysis ignores data order
• Believing it only works with numbers, not dates
• Assuming it automatically cleans missing data

2. Which of the following is the correct way to convert a column named 'date' to datetime in pandas?

easy

A. df['date'] = convert_to_date(df['date'])

B. df['date'] = datetime(df['date'])

C. df['date'] = pd.datetime(df['date'])

D. df['date'] = pd.to_datetime(df['date'])

Solution

1. Step 1: Recall pandas datetime conversion function

   pandas uses pd.to_datetime() to convert columns to datetime format.

2. Step 2: Check each option for correctness

   df['date'] = pd.to_datetime(df['date']) uses the correct function and syntax. Options B, C, and D use invalid or non-existent functions.

3. Final Answer:

   df['date'] = pd.to_datetime(df['date']) -> Option D

4. Quick Check:

   Use pd.to_datetime() for date conversion = A [OK]

Hint: Use pd.to_datetime() to convert date columns [OK]

Common Mistakes:

• Using datetime() instead of pd.to_datetime()
• Trying pd.datetime() which does not exist
• Assuming a function named convert_to_date exists

3. Given the following code, what will be the output?

import pandas as pd

data = {'date': ['2023-01-01', '2023-01-02', '2023-01-08'], 'value': [10, 20, 30]}
df = pd.DataFrame(data)
df['date'] = pd.to_datetime(df['date'])
df.set_index('date', inplace=True)
weekly_sum = df.resample('W').sum()
print(weekly_sum)

medium

A. A DataFrame with weekly sums: 2023-01-01 sum=10, 2023-01-02 to 2023-01-08 sum=50

B. A DataFrame with daily sums for each date

C. An error because resample needs a column, not index

D. Empty DataFrame because dates are not sorted

Solution

1. Step 1: Understand resample with weekly frequency

   Resampling with 'W' groups data by week ending on Sunday. 2023-01-01 (Sunday) falls in week ending 2023-01-01, 2023-01-02 (Monday) to 2023-01-08 (Sunday) fall in week ending 2023-01-08.

2. Step 2: Calculate sums per week

   First week sum: 10; second week sum: 20 + 30 = 50.

3. Final Answer:

   A DataFrame with weekly sums: 2023-01-01 sum=10, 2023-01-02 to 2023-01-08 sum=50 -> Option A

4. Quick Check:

   Weekly resample sums values correctly = C [OK]

Hint: Resample groups by time; sum adds values per period [OK]

Common Mistakes:

• Thinking resample works on columns instead of index
• Assuming daily sums instead of weekly
• Believing resample requires sorted dates (pandas sorts automatically)

4. What is wrong with this code snippet for time-based analysis?

import pandas as pd
data = {'date': ['2023-01-01', '2023-01-02'], 'value': [5, 10]}
df = pd.DataFrame(data)
df['date'] = pd.to_datetime(df['date'])
df.set_index('date')
weekly_avg = df.resample('W').mean()
print(weekly_avg)

medium

A. resample('W') cannot be used with mean().

B. pd.to_datetime is used incorrectly.

C. The index is not set inplace, so resample fails on original index.

D. DataFrame is missing a 'value' column.

Solution

1. Step 1: Check index setting

   df.set_index('date') returns a new DataFrame but does not change df unless inplace=True is used or reassigned.

2. Step 2: Understand resample requirement

   Resample works on the datetime index. Since index was not set inplace, df still has default integer index, causing wrong grouping.

3. Final Answer:

   The index is not set inplace, so resample fails on original index. -> Option C

4. Quick Check:

   set_index needs inplace=True or reassignment = A [OK]

Hint: Use inplace=True or assign set_index result [OK]

Common Mistakes:

• Forgetting inplace=True when setting index
• Thinking resample works without datetime index
• Assuming mean() is invalid with resample

5. You have daily sales data for a year. You want to find monthly sales trends but notice some days are missing. Which approach best reveals accurate monthly trends?

hard

A. Convert dates to datetime, set as index, then resample monthly with sum(), filling missing days with 0.

B. Ignore missing days and calculate monthly sum directly on raw data.

C. Convert dates to string and group by month substring, then sum values.

D. Use daily data without conversion and average all values.

Solution

1. Step 1: Prepare data with datetime index

   Convert the date column to datetime and set it as the DataFrame index for time-based operations.

2. Step 2: Resample monthly and handle missing days

   Use resample('M').sum() to aggregate monthly sales. Fill missing days with 0 to avoid undercounting.

3. Step 3: Avoid incorrect grouping methods

   Grouping by string substrings or ignoring missing days can cause inaccurate trends.

4. Final Answer:

   Convert dates to datetime, set as index, then resample monthly with sum(), filling missing days with 0. -> Option A

5. Quick Check:

   Proper datetime index + resample + fill missing = B [OK]

Hint: Set datetime index, resample monthly, fill missing with zero [OK]

Common Mistakes:

• Grouping by string instead of datetime index
• Ignoring missing days causing wrong sums
• Not converting dates to datetime before resampling