Date feature extraction in Data Analysis Python - Time & Space Complexity

Time Complexity: Date feature extraction

O(n)

Understanding Time Complexity

We want to understand how the time to extract parts of dates grows as we handle more data.

How does the work change when we have more dates to process?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

dates = pd.Series(pd.date_range('2023-01-01', periods=1000))

# Extract year, month, day from each date
features = pd.DataFrame({
    'year': dates.dt.year,
    'month': dates.dt.month,
    'day': dates.dt.day
})

This code creates a list of dates and extracts year, month, and day parts into a new table.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Extracting year, month, and day from each date in the list.
• How many times: Once for each date in the input series.

How Execution Grows With Input

Each date is processed individually to get its parts, so the work grows directly with the number of dates.

Input Size (n)	Approx. Operations
10	About 30 (3 parts x 10 dates)
100	About 300 (3 parts x 100 dates)
1000	About 3000 (3 parts x 1000 dates)

Pattern observation: The total work increases steadily as we add more dates, roughly multiplying by the number of dates.

Final Time Complexity

Time Complexity: O(n)

This means the time to extract date parts grows in a straight line with the number of dates.

Common Mistake

[X] Wrong: "Extracting multiple parts from dates takes the same time no matter how many dates there are."

[OK] Correct: Each date must be processed separately, so more dates mean more work and more time.

Interview Connect

Understanding how data size affects processing time helps you explain your code choices clearly and shows you think about efficiency.

Self-Check

"What if we extracted more features like hour, minute, and second? How would the time complexity change?"

Practice

1. What does the dt.year attribute do when applied to a pandas datetime column?

easy

A. Extracts the year part from each date in the column

B. Extracts the day of the week as a number

C. Converts the date to a string format

D. Calculates the difference between dates

Solution

1. Step 1: Understand pandas datetime accessor

   The dt accessor allows extraction of date parts from datetime columns.

2. Step 2: Identify what dt.year extracts

   dt.year extracts the year component from each datetime value.

3. Final Answer:

   Extracts the year part from each date in the column -> Option A

4. Quick Check:

   dt.year = year extraction [OK]

Hint: Remember dt.year gives the year number from dates [OK]

Common Mistakes:

• Confusing dt.year with dt.day or dt.month
• Thinking dt.year converts dates to strings
• Assuming dt.year calculates date differences

2. Which of the following is the correct syntax to extract the month from a pandas datetime column named df['date']?

easy

A. df['date'].month

B. df['date'].dt.month

C. df['date'].get_month()

D. df['date'].extract('month')

Solution

1. Step 1: Recall pandas datetime accessor usage

   To extract date parts, pandas requires the dt accessor before the attribute.

2. Step 2: Identify correct syntax for month extraction

   The correct syntax is df['date'].dt.month. Other options are invalid methods or missing dt.

3. Final Answer:

   df['date'].dt.month -> Option B

4. Quick Check:

   Use dt.month to get month [OK]

Hint: Use .dt before date parts like month or day [OK]

Common Mistakes:

• Omitting .dt accessor
• Using non-existent methods like get_month()
• Trying to extract with extract('month')

3. Given the following code, what will be the output?

import pandas as pd
dates = pd.to_datetime(['2023-01-15', '2023-07-04', '2023-12-31'])
df = pd.DataFrame({'date': dates})
df['day_of_week'] = df['date'].dt.dayofweek
print(df['day_of_week'].tolist())

medium

A. [6, 1, 0]

B. [0, 1, 6]

C. [6, 2, 6]

D. [6, 1, 6]

Solution

1. Step 1: Convert dates to day of week numbers

   Using dt.dayofweek, Monday=0 ... Sunday=6. Dates: 2023-01-15 is Sunday (6), 2023-07-04 is Tuesday (1), 2023-12-31 is Sunday (6).

2. Step 2: List the dayofweek values

   The list is [6, 1, 6].

3. Final Answer:

   [6, 1, 6] -> Option D

4. Quick Check:

   Sunday=6, Tuesday=1, Sunday=6 [OK]

Hint: Remember Monday=0, Sunday=6 for dayofweek [OK]

Common Mistakes:

• Confusing dayofweek with weekday names
• Assuming Sunday=0 instead of 6
• Mixing up dayofweek and day_name()

4. What is wrong with this code snippet to extract the hour from a datetime column df['timestamp']?

df['hour'] = df['timestamp'].hour

medium

A. Missing .dt accessor before .hour

B. The column must be converted to string first

C. The .hour attribute does not exist in pandas

D. The code should use .time.hour instead

Solution

1. Step 1: Check pandas datetime attribute access

   Datetime properties require the dt accessor to access parts like hour.

2. Step 2: Identify missing .dt in code

   The code uses df['timestamp'].hour which is invalid; it should be df['timestamp'].dt.hour.

3. Final Answer:

   Missing .dt accessor before .hour -> Option A

4. Quick Check:

   Use .dt.hour to extract hour [OK]

Hint: Always use .dt before date parts like hour [OK]

Common Mistakes:

• Trying to access .hour directly on Series
• Converting datetime to string unnecessarily
• Using .time.hour which is invalid in pandas

5. You have a pandas DataFrame df with a datetime column 'order_date'. You want to create a new column 'is_weekend' that is True if the order was placed on Saturday or Sunday, and False otherwise. Which code snippet correctly achieves this?

hard

A. df['is_weekend'] = df['order_date'].dt.dayofweek == 6 or 5

B. df['is_weekend'] = df['order_date'].dt.weekday in [5,6]

C. df['is_weekend'] = df['order_date'].dt.dayofweek >= 5

D. df['is_weekend'] = df['order_date'].dt.day_name() in ['Saturday', 'Sunday']

Solution

1. Step 1: Understand dayofweek values for weekend

   In pandas, dt.dayofweek returns 0 for Monday through 6 for Sunday. Weekend days are 5 (Saturday) and 6 (Sunday).

2. Step 2: Check which code correctly marks weekend

   df['is_weekend'] = df['order_date'].dt.dayofweek >= 5 uses df['order_date'].dt.dayofweek >= 5, which is True for 5 and 6, correctly marking weekends. Other options use invalid syntax or incorrect logic.

3. Final Answer:

   df['is_weekend'] = df['order_date'].dt.dayofweek >= 5 -> Option C

4. Quick Check:

   Weekend days are dayofweek >= 5 [OK]

Hint: Weekend days have dayofweek 5 or 6, use >= 5 [OK]

Common Mistakes:

• Using 'in' without vectorized methods
• Using 'or' instead of bitwise operators
• Confusing dayofweek numbers or names