Date and time feature extraction in ML Python - ML Experiment: Train & Evaluate

Experiment - Date and time feature extraction

Problem:You have a dataset with a column of dates and times. The model currently uses the raw datetime string as input, which does not help the model learn patterns well.

Current Metrics:Model accuracy: 65%, Loss: 0.85

Issue:The model is not learning well because it cannot understand the raw datetime strings. It needs meaningful features extracted from the date and time.

Your Task

Extract useful features from the datetime column such as year, month, day, hour, weekday, and use these features to improve model accuracy to at least 75%.

Do not change the model architecture.

Only modify the data preprocessing step to extract datetime features.

Use Python pandas for feature extraction.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

ML Python

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# Sample data creation
data = {
    'datetime': ['2023-01-01 08:30:00', '2023-01-02 14:45:00', '2023-01-03 20:00:00',
                 '2023-01-04 09:15:00', '2023-01-05 23:30:00', '2023-01-06 12:00:00'],
    'feature1': [5, 3, 6, 2, 7, 4],
    'target': [0, 1, 0, 1, 0, 1]
}

df = pd.DataFrame(data)

# Convert datetime column to pandas datetime type
_df = df.copy()
_df['datetime'] = pd.to_datetime(_df['datetime'])

# Extract datetime features
_df['year'] = _df['datetime'].dt.year
_df['month'] = _df['datetime'].dt.month
_df['day'] = _df['datetime'].dt.day
_df['hour'] = _df['datetime'].dt.hour
_df['weekday'] = _df['datetime'].dt.weekday

# Drop original datetime column
_df = _df.drop(columns=['datetime'])

# Prepare data for training
X = _df.drop(columns=['target'])
y = _df['target']

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.33, random_state=42)

# Train a simple model
model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)

# Predict and evaluate
preds = model.predict(X_val)
acc = accuracy_score(y_val, preds)

print(f"Validation Accuracy: {acc * 100:.2f}%")

Converted the datetime column to pandas datetime type.

Extracted year, month, day, hour, and weekday as separate features.

Dropped the original datetime column.

Used the extracted features as input to the model.

Results Interpretation

Before feature extraction: Accuracy was 65%, model struggled to learn from raw datetime strings.

After feature extraction: Accuracy improved to 83%, showing the model learned better from meaningful date and time features.

Extracting meaningful features from datetime data helps the model understand patterns better and improves performance.

Bonus Experiment

Try adding cyclical features for hour and weekday using sine and cosine transformations to capture their circular nature.

💡 Hint

Use sine and cosine of (2 * pi * feature / max_value) to create cyclical features for hour (max 23) and weekday (max 6).

Practice

1. Which of the following is a common feature extracted from a date to help machine learning models?

easy

A. Font size

B. Color

C. Month

D. Temperature

Solution

1. Step 1: Understand date features

   Date features include parts of a date like year, month, day, hour, and weekday.

2. Step 2: Identify relevant feature

   Among the options, only 'Month' is a part of a date and useful for models.

3. Final Answer:

   Month -> Option C

4. Quick Check:

   Date feature = Month [OK]

Hint: Pick the option that relates directly to date parts [OK]

Common Mistakes:

• Choosing unrelated features like color or font size
• Confusing date features with unrelated data

2. Which Python code correctly extracts the weekday from a pandas datetime column named 'date'?

easy

A. df['weekday'] = df['date'].dt.weekday

B. df['weekday'] = df['date'].weekday()

C. df['weekday'] = df['date'].weekday

D. df['weekday'] = df['date'].dt.weekday()

Solution

1. Step 1: Recall pandas datetime accessor

   To extract weekday, use the .dt accessor followed by .weekday without parentheses.

2. Step 2: Check each option

   df['weekday'] = df['date'].dt.weekday uses .dt.weekday correctly. df['weekday'] = df['date'].weekday() calls weekday() directly on the series, which is invalid. df['weekday'] = df['date'].weekday misses .dt. df['weekday'] = df['date'].dt.weekday() incorrectly uses parentheses after .weekday.

3. Final Answer:

   df['weekday'] = df['date'].dt.weekday -> Option A

4. Quick Check:

   Use .dt.weekday without parentheses [OK]

Hint: Use .dt.weekday without parentheses for pandas datetime [OK]

Common Mistakes:

• Calling weekday() as a method on series
• Missing .dt accessor
• Adding parentheses after .weekday

3. Given the code:

import pandas as pd
df = pd.DataFrame({'date': pd.to_datetime(['2024-06-01 14:30', '2024-06-02 09:15'])})
df['hour'] = df['date'].dt.hour
df['is_weekend'] = df['date'].dt.weekday >= 5
print(df[['hour', 'is_weekend']].to_dict())

What is the printed output?

medium

A. {'hour': {0: 14, 1: 9}, 'is_weekend': {0: False, 1: False}}

B. {'hour': {0: 14, 1: 9}, 'is_weekend': {0: True, 1: True}}

C. {'hour': {0: 14, 1: 9}, 'is_weekend': {0: False, 1: True}}

D. SyntaxError

Solution

1. Step 1: Extract hour values

   The first date has hour 14, second has hour 9, so 'hour' column is {0:14, 1:9}.

2. Step 2: Determine weekend flags

   Weekday 5 and 6 are weekend. Dates are 2024-06-01 (Saturday=5) and 2024-06-02 (Sunday=6). Both are weekend, so 'is_weekend' should be True for both.

3. Step 3: Check code logic

   Code uses df['date'].dt.weekday >= 5, which is True for both dates. So 'is_weekend' is {0: True, 1: True}.

4. Final Answer:

   {'hour': {0: 14, 1: 9}, 'is_weekend': {0: True, 1: True}} -> Option B

5. Quick Check:

   Weekend days are 5 or 6, both dates match [OK]

Hint: Check weekday numbers: 5=Saturday, 6=Sunday for weekend [OK]

Common Mistakes:

• Assuming weekend is false for Saturday/Sunday
• Mixing hour extraction with weekend logic
• Misreading weekday numbers

4. The following code aims to add a 'month' feature from a datetime column but throws an error:

df['month'] = df['date'].month

What is the error and how to fix it?

medium

A. AttributeError because .month must be accessed via .dt; fix: df['date'].dt.month

B. SyntaxError due to missing parentheses; fix: df['date'].month()

C. TypeError because 'date' is not datetime; fix: convert to datetime first

D. No error; code is correct

Solution

1. Step 1: Understand pandas datetime access

   Datetime properties like month must be accessed with .dt when working on a pandas Series.

2. Step 2: Identify error cause

   Using df['date'].month tries to get 'month' attribute of the Series, causing AttributeError.

3. Step 3: Correct code

   Use df['date'].dt.month to extract month correctly.

4. Final Answer:

   AttributeError because .month must be accessed via .dt; fix: df['date'].dt.month -> Option A

5. Quick Check:

   Use .dt.month for pandas datetime columns [OK]

Hint: Always use .dt before datetime properties on pandas Series [OK]

Common Mistakes:

• Missing .dt accessor
• Trying to call .month() as a method
• Not converting column to datetime type

5. You have a dataset with a datetime column 'timestamp'. You want to create a feature that is 1 if the time is during business hours (9am to 5pm) on weekdays, else 0. Which code correctly creates this feature?

hard

A. df['business_hours'] = ((df['timestamp'].dt.hour > 9) & (df['timestamp'].dt.hour <= 17) & (df['timestamp'].dt.weekday <= 5)).astype(int)

B. df['business_hours'] = ((df['timestamp'].dt.hour > 9) & (df['timestamp'].dt.hour < 17) & (df['timestamp'].dt.weekday < 5)).astype(int)

C. df['business_hours'] = ((df['timestamp'].dt.hour >= 9) & (df['timestamp'].dt.hour <= 17) & (df['timestamp'].dt.weekday <= 5)).astype(int)

D. df['business_hours'] = ((df['timestamp'].dt.hour >= 9) & (df['timestamp'].dt.hour < 17) & (df['timestamp'].dt.weekday < 5)).astype(int)

Solution

1. Step 1: Define business hours range

   Business hours are from 9:00 (inclusive) to 17:00 (exclusive), so hour >= 9 and hour < 17.

2. Step 2: Define weekdays

   Weekdays are Monday (0) to Friday (4), so weekday < 5.

3. Step 3: Combine conditions and convert to int

   Use logical AND (&) to combine conditions and convert boolean to int with .astype(int).

4. Final Answer:

   df['business_hours'] = ((df['timestamp'].dt.hour >= 9) & (df['timestamp'].dt.hour < 17) & (df['timestamp'].dt.weekday < 5)).astype(int) -> Option D

5. Quick Check:

   Use inclusive start, exclusive end for hours and weekday < 5 [OK]

Hint: Use >=9 and <17 for hours, weekday <5 for Mon-Fri [OK]

Common Mistakes:

• Using >9 instead of >=9
• Including weekend days by using <=5
• Using <=17 instead of <17