Why engineered features improve models in ML Python

Introduction

Engineered features help models learn better by giving them clearer and more useful information from the data.

When raw data is too complex or noisy for the model to understand well.

When you want to highlight important patterns or relationships in the data.

When the model's accuracy is low and you want to improve it by adding meaningful inputs.

When you have domain knowledge that can create new helpful data points.

When you want to reduce the amount of data the model needs to learn from.

Syntax

ML Python

No fixed syntax because feature engineering is about creating new data columns or transforming existing ones using code or tools.

Feature engineering often uses simple operations like math, grouping, or combining columns.

It can be done using programming languages like Python with libraries such as pandas.

Examples

Create a new feature by squaring the 'age' column to capture non-linear effects.

ML Python

df['age_squared'] = df['age'] ** 2

Create a new feature that divides income by family size to get a per-person value.

ML Python

df['income_per_person'] = df['income'] / df['family_size']

Create a binary feature indicating if a day is on the weekend.

ML Python

df['is_weekend'] = df['day_of_week'].apply(lambda x: 1 if x in ['Saturday', 'Sunday'] else 0)

Sample Model

This example shows how adding a new feature 'income_per_person' can help the model predict house prices better by lowering the error.

ML Python

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

# Sample data
data = {'age': [25, 32, 47, 51, 62],
         'income': [50000, 60000, 80000, 90000, 120000],
         'family_size': [3, 4, 2, 5, 3],
         'house_price': [200000, 250000, 320000, 360000, 400000]}

# Create DataFrame
df = pd.DataFrame(data)

# Feature engineering: create income per person
df['income_per_person'] = df['income'] / df['family_size']

# Prepare features and target
X = df[['age', 'income', 'family_size']]
X_eng = df[['age', 'income', 'family_size', 'income_per_person']]
y = df['house_price']

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
X_eng_train, X_eng_test, y_train_eng, y_test_eng = train_test_split(X_eng, y, random_state=42)

# Train model without engineered feature
model = LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
mse_without = mean_squared_error(y_test, y_pred)

# Train model with engineered feature
model_eng = LinearRegression()
model_eng.fit(X_eng_train, y_train_eng)
y_pred_eng = model_eng.predict(X_eng_test)
mse_with = mean_squared_error(y_test_eng, y_pred_eng)

print(f"MSE without engineered feature: {mse_without:.2e}")
print(f"MSE with engineered feature: {mse_with:.2e}")

Important Notes

Feature engineering can greatly improve model results but requires understanding the data well.

Sometimes adding too many features can confuse the model, so choose features carefully.

Try simple transformations first before complex ones.

Summary

Engineered features give models clearer, more useful information.

They help models find patterns that raw data might hide.

Good feature engineering can improve accuracy and reduce errors.

Practice

1. Why do engineered features often help machine learning models perform better?

easy

A. They remove the need for training the model.

B. They make the model run faster by reducing the number of layers.

C. They provide clearer and more useful information for the model to learn from.

D. They increase the size of the dataset automatically.

Solution

1. Step 1: Understand the role of features in machine learning

   Features are the pieces of information the model uses to find patterns and make predictions.

2. Step 2: Recognize how engineered features improve clarity

   Engineered features transform raw data into clearer, more meaningful forms that help the model learn better.

3. Final Answer:

   They provide clearer and more useful information for the model to learn from. -> Option C

4. Quick Check:

   Clear features = Better learning [OK]

Hint: Engineered features clarify data meaning for models [OK]

Common Mistakes:

• Thinking engineered features speed up training by reducing layers
• Believing engineered features increase dataset size automatically
• Assuming engineered features remove need for training

2. Which of the following is the correct way to create a new feature called age_group from an age column in Python using pandas?

easy

A. df['age_group'] = df['age'].mean()

B. df['age_group'] = df['age'] > 30

C. df['age_group'] = df['age'].sum()

D. df['age_group'] = df['age'].apply(lambda x: 'young' if x < 30 else 'old')

Solution

1. Step 1: Identify how to create categorical features from numeric data

   Using apply with a function lets us assign categories like 'young' or 'old' based on age.

2. Step 2: Check each option for correctness

   df['age_group'] = df['age'].apply(lambda x: 'young' if x < 30 else 'old') uses apply with a lambda function to create age_group correctly. df['age_group'] = df['age'] > 30 creates a boolean, not a group. The sum and mean options compute sums or means, not groups.

3. Final Answer:

   df['age_group'] = df['age'].apply(lambda x: 'young' if x < 30 else 'old') -> Option D

4. Quick Check:

   Use apply + lambda for new categorical features [OK]

Hint: Use apply with lambda for conditional feature creation [OK]

Common Mistakes:

• Using sum or mean instead of conditional logic
• Creating boolean instead of categorical feature
• Not using apply or map for transformation

3. Given this code snippet, what will be the output of print(df) after feature engineering?

import pandas as pd
df = pd.DataFrame({'temp_c': [0, 20, 30]})
df['temp_f'] = df['temp_c'] * 9/5 + 32
print(df)

medium

A. temp_c temp_f
0 0 32.0
1 20 68.0
2 30 86.0

B. temp_c temp_f
0 0 0.0
1 20 20.0
2 30 30.0

C. temp_c temp_f
0 0 32
1 20 68
2 30 86

D. Error: Cannot multiply series by float

Solution

1. Step 1: Understand the temperature conversion formula

   Fahrenheit = Celsius * 9/5 + 32. The code applies this formula to each value in temp_c.

2. Step 2: Calculate the converted values

   For 0°C: 0*9/5+32=32.0; for 20°C: 20*9/5+32=68.0; for 30°C: 30*9/5+32=86.0. The values are floats.

3. Final Answer:

   temp_c temp_f 0 0 32.0 1 20 68.0 2 30 86.0 -> Option A

4. Quick Check:

   Correct formula applied element-wise = temp_c temp_f 0 0 32.0 1 20 68.0 2 30 86.0 [OK]

Hint: Apply formulas element-wise for new numeric features [OK]

Common Mistakes:

• Confusing Celsius and Fahrenheit formulas
• Expecting integer instead of float results
• Thinking pandas cannot multiply series by float

4. You wrote this code to create a new feature is_adult but it gives wrong results. What is the bug?

df['is_adult'] = df['age'] > '18'

medium

A. Comparing numeric age to string '18' causes incorrect results.

B. The operator > cannot be used in pandas.

C. The new feature should be named adult_flag instead.

D. You must use double equals == for comparison.

Solution

1. Step 1: Identify data type mismatch in comparison

   The code compares numeric age values to a string '18', which leads to wrong boolean results.

2. Step 2: Correct the comparison by using a numeric value

   Replace '18' (string) with 18 (integer) to compare numbers properly.

3. Final Answer:

   Comparing numeric age to string '18' causes incorrect results. -> Option A

4. Quick Check:

   Match data types in comparisons [OK]

Hint: Compare numbers to numbers, not strings [OK]

Common Mistakes:

• Using string instead of numeric for comparison
• Thinking > operator is invalid in pandas
• Confusing == with > for this logic

5. You have a dataset with raw timestamps and want to improve your model predicting sales. Which engineered feature is most likely to help the model find useful patterns?

hard

A. Converting timestamps to strings without changes.

B. Extracting the hour of day and day of week from the timestamp.

C. Removing all timestamp data to reduce complexity.

D. Replacing timestamps with random numbers.

Solution

1. Step 1: Understand what useful information timestamps hold

   Timestamps contain time details that can reveal patterns like busy hours or weekdays.

2. Step 2: Identify which feature extraction helps models

   Extracting hour and day of week turns raw timestamps into meaningful features that models can use to detect trends.

3. Final Answer:

   Extracting the hour of day and day of week from the timestamp. -> Option B

4. Quick Check:

   Meaningful time features improve pattern detection [OK]

Hint: Turn raw timestamps into time parts like hour/day [OK]

Common Mistakes:

• Keeping timestamps as strings without extraction
• Removing timestamps losing useful info
• Replacing timestamps with random data