Feature engineering basics in Pandas - Deep Dive

Overview - Feature engineering basics

What is it?

Feature engineering is the process of creating new input variables or modifying existing ones to help a machine learning model learn better. It involves transforming raw data into meaningful features that highlight important patterns. This step is crucial because models can only learn from the features they are given. Without good features, even the best algorithms struggle to make accurate predictions.

Why it matters

Without feature engineering, models often miss important signals hidden in raw data, leading to poor predictions. It solves the problem of making data understandable and useful for machines. Imagine trying to solve a puzzle with missing or unclear pieces; feature engineering fills in those gaps and sharpens the picture. This improves model accuracy, reduces training time, and helps uncover insights that raw data alone cannot reveal.

Where it fits

Before learning feature engineering, you should understand basic data handling with pandas and simple statistics. After mastering feature engineering, you can move on to model building, tuning, and evaluation. It sits between data cleaning and model training in the data science workflow.

Mental Model

Core Idea

Feature engineering is like crafting the right questions from raw data so a model can find the best answers.

Think of it like...

Think of raw data as a block of marble and feature engineering as the sculptor chiseling it into a statue. The sculptor removes unnecessary parts and shapes the marble to reveal the form inside. Similarly, feature engineering shapes raw data into useful features that reveal patterns for the model.

Raw Data ──▶ Feature Engineering ──▶ Model Training
  │                 │
  │                 └─ New Features (columns) created
  └─ Original Data   └─ Existing Features transformed

Build-Up - 7 Steps

1

Foundation

Understanding raw data structure

Concept: Learn how raw data looks and what types of columns it contains.

Using pandas, load a simple dataset and explore its columns, data types, and missing values. For example, load a CSV file and use df.head(), df.info(), and df.describe() to understand the data.

Result

You see the dataset's shape, column names, data types, and summary statistics.

Understanding the raw data structure is essential because feature engineering depends on knowing what data you have and its quality.

2

Foundation

Basic feature transformations

Concept: Learn simple ways to change existing features to improve their usefulness.

Try converting categorical columns to numbers using pandas' .map() or .astype('category').cat.codes. Create new columns by combining or modifying existing ones, like adding two columns or extracting the year from a date.

Result

New or transformed columns appear in the DataFrame, ready for modeling.

Simple transformations can reveal hidden patterns and make data easier for models to understand.

3

Intermediate

Handling missing values smartly

🤔Before reading on: do you think dropping rows with missing data is always the best choice? Commit to your answer.

Concept: Learn different strategies to fill or handle missing data instead of just dropping it.

Use pandas methods like .fillna() with mean, median, or a fixed value. Explore creating a new boolean feature indicating if data was missing. This preserves information and avoids losing valuable data.

Result

The dataset has no missing values, and sometimes new features indicate missingness.

Knowing how to handle missing data prevents loss of information and can improve model performance by signaling where data was incomplete.

4

Intermediate

Creating interaction features

🤔Before reading on: do you think combining two features always improves model accuracy? Commit to your answer.

Concept: Learn to create new features by combining two or more existing features to capture relationships.

Multiply, add, or concatenate columns to create interaction features. For example, multiply 'age' by 'income' to capture combined effects. Use pandas operations to create these new columns.

Result

New interaction features appear in the DataFrame, potentially capturing complex patterns.

Interaction features can reveal relationships between variables that single features miss, helping models learn better.

5

Intermediate

Feature scaling and normalization

Concept: Learn why and how to scale features to a common range or distribution.

Use pandas or sklearn to scale features with Min-Max scaling or Standardization (subtract mean, divide by std). This helps models that are sensitive to feature scale, like distance-based algorithms.

Result

Features are transformed to similar scales, improving model training stability.

Scaling prevents features with large values from dominating the model and helps algorithms converge faster.

6

Advanced

Encoding categorical variables effectively

🤔Before reading on: do you think one-hot encoding is always the best way to encode categories? Commit to your answer.

Concept: Explore different encoding methods beyond one-hot, like target encoding or frequency encoding.

Use pandas to implement one-hot encoding with pd.get_dummies(). Then try target encoding by replacing categories with the mean target value. Compare when each method works better.

Result

Categorical variables are transformed into numeric features suitable for models, with different trade-offs.

Choosing the right encoding method balances model complexity and overfitting risk, improving prediction quality.

7

Expert

Feature engineering automation and pitfalls

🤔Before reading on: do you think automated feature engineering always outperforms manual crafting? Commit to your answer.

Concept: Understand tools that automate feature creation and their limitations.

Explore libraries like Featuretools that generate many features automatically. Learn that while automation saves time, it can create noisy or redundant features. Manual review and domain knowledge remain crucial.

Result

Automated features expand the dataset but require careful selection to avoid overfitting.

Knowing automation limits prevents blindly trusting generated features and encourages combining human insight with tools.

Under the Hood

Feature engineering works by transforming raw data into numerical arrays that machine learning algorithms can process. Internally, pandas stores data in tables with typed columns. Transformations create new columns or modify existing ones, changing the data representation. These changes affect how algorithms calculate distances, correlations, or splits during training, directly impacting model learning.

Why designed this way?

Feature engineering evolved because raw data is often messy, incomplete, or not in a form that algorithms understand. Early machine learning struggled with raw inputs, so transforming data into meaningful features became essential. The design balances flexibility (many transformation types) with efficiency (fast pandas operations) to handle large datasets.

┌─────────────┐      ┌─────────────────────┐      ┌───────────────┐
│ Raw Dataset │─────▶│ Feature Engineering │─────▶│ Transformed   │
│ (pandas DF) │      │ (transformations)   │      │ Dataset       │
└─────────────┘      └─────────────────────┘      └───────────────┘
       │                      │                          │
       │                      │                          │
       ▼                      ▼                          ▼
  Columns with          New columns created        Numeric arrays
  raw values            or existing columns        ready for model
                        modified

Myth Busters - 4 Common Misconceptions

Quick: Is dropping all rows with missing data always the best way to handle missing values? Commit to yes or no.

Common Belief: Dropping rows with missing data is the safest and best way to handle missing values.

Reality: Dropping rows can remove too much data and lose important information. Filling missing values or creating indicators often works better.

Why it matters: Dropping data unnecessarily reduces training size and can bias the model, leading to worse predictions.

Quick: Do you think one-hot encoding is always the best method for categorical variables? Commit to yes or no.

Common Belief: One-hot encoding is the best and only way to convert categorical variables for models.

Reality: One-hot encoding can create too many features and cause overfitting. Other methods like target encoding or frequency encoding can be better in some cases.

Why it matters: Using one-hot encoding blindly can slow training and reduce model generalization.

Quick: Does scaling features always improve model performance? Commit to yes or no.

Common Belief: Scaling features always improves model accuracy regardless of algorithm.

Reality: Scaling helps some algorithms (like k-NN, SVM) but is unnecessary for tree-based models like random forests.

Why it matters: Unnecessary scaling wastes time and can confuse beginners about when it matters.

Quick: Do automated feature engineering tools always produce better features than manual work? Commit to yes or no.

Common Belief: Automated feature engineering tools always outperform manual feature creation.

Reality: Automation can create many irrelevant or redundant features, requiring human judgment to select the best ones.

Why it matters: Relying solely on automation can lead to overfitting and poor model interpretability.

Expert Zone

1

Feature interactions can explode feature space size; careful selection or dimensionality reduction is needed to avoid overfitting.

2

Encoding categorical variables with high cardinality requires balancing detail and model complexity; sometimes hashing tricks are used.

3

Missing value indicators can themselves be predictive features, signaling data collection issues or special cases.

When NOT to use

Feature engineering is less critical for deep learning models on large datasets where raw data can be fed directly. In such cases, automated feature extraction layers replace manual engineering. Also, for very small datasets, complex feature engineering can cause overfitting; simpler features or domain knowledge may be better.

Production Patterns

In production, feature engineering pipelines are automated and version-controlled to ensure consistency between training and serving. Feature stores are used to manage and serve features efficiently. Real-time feature computation and monitoring are common to maintain model accuracy over time.

Connections

Data Cleaning

Builds-on

Good feature engineering depends on clean data; understanding data cleaning helps avoid garbage-in-garbage-out problems.

Signal Processing

Similar pattern

Both transform raw inputs into meaningful signals; feature engineering extracts patterns like filters extract frequencies.

Cooking Recipes

Analogy in process

Just as cooking transforms raw ingredients into a tasty dish, feature engineering transforms raw data into useful inputs for models.

Common Pitfalls

#1 Dropping all rows with missing data without considering alternatives.

Wrong approach:

df_clean = df.dropna()

Correct approach:

df['col_filled'] = df['col'].fillna(df['col'].mean())
df['col_missing'] = df['col'].isna().astype(int)

Root cause:

Misunderstanding that missing data always means bad data, ignoring that missingness can carry information.

#2 Using one-hot encoding on high-cardinality categorical variables causing too many features.

Wrong approach:

df_encoded = pd.get_dummies(df['category_col'])

Correct approach:

freq = df['category_col'].value_counts(normalize=True)
df['category_freq'] = df['category_col'].map(freq)

Root cause:

Not realizing that one-hot encoding creates many sparse columns, which can slow training and cause overfitting.

#3 Scaling features for tree-based models where it is unnecessary.

Wrong approach:

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
df_scaled = scaler.fit_transform(df[['feature1', 'feature2']])

Correct approach:

# No scaling needed for tree models
model.fit(df[['feature1', 'feature2']], target)

Root cause:

Assuming all models require scaled features without understanding algorithm differences.

Key Takeaways

Feature engineering transforms raw data into meaningful inputs that help models learn better.

Understanding your data's structure and quality is essential before creating features.

Different feature transformations suit different data types and models; there is no one-size-fits-all.

Handling missing data thoughtfully and encoding categorical variables properly can greatly improve model accuracy.

Automation can help but human insight and domain knowledge remain critical for effective feature engineering.

Practice

1. What is the main purpose of feature engineering in pandas?

easy

A. To create new useful columns from existing data

B. To delete unnecessary rows from the dataset

C. To change the data type of columns

D. To visualize data using charts

Solution

1. Step 1: Understand feature engineering definition

   Feature engineering means creating new columns that help models learn better.

2. Step 2: Compare options with definition

   Only To create new useful columns from existing data matches the purpose of creating new useful columns from existing data.

3. Final Answer:

   To create new useful columns from existing data -> Option A

4. Quick Check:

   Feature engineering = create new columns [OK]

Hint: Feature engineering = making new columns from old data [OK]

Common Mistakes:

• Confusing feature engineering with data cleaning
• Thinking it only changes data types
• Mixing it up with data visualization

2. Which of the following is the correct way to create a new column total as sum of columns 2a and 2b in a pandas DataFrame df?

easy

A. df['total'] <- df['2a'] + df['2b']

B. df.total = df.2a + df.2b

C. df['total'] == df['2a'] + df['2b']

D. df['total'] = df['2a'] + df['2b']

Solution

1. Step 1: Recall pandas syntax for new column

   Use df['new_col'] = expression to create new columns.

2. Step 2: Check each option

   df['total'] = df['2a'] + df['2b'] uses correct syntax. df.total = df.2a + df.2b misses brackets, A uses wrong operator, D uses comparison operator.

3. Final Answer:

   df['total'] = df['2a'] + df['2b'] -> Option D

4. Quick Check:

   New column = df['col'] = ... [OK]

Hint: Use df['col'] = ... to add new columns [OK]

Common Mistakes:

• Using == instead of = for assignment
• Using <- which is not Python syntax
• Accessing columns without brackets

3. Given the DataFrame df below, what will be the output of df['age_group'] after this code?

import pandas as pd
df = pd.DataFrame({'age': [15, 22, 37, 45]})
df['age_group'] = pd.cut(df['age'], bins=[0, 18, 35, 60], labels=['child', 'young_adult', 'adult'])

medium

A. [child, young_adult, young_adult, adult]

B. [child, child, young_adult, adult]

C. [child, young_adult, adult, adult]

D. [child, young_adult, adult, young_adult]

Solution

1. Step 1: Understand pd.cut bins and labels

   Bins: 0-18 = child, 18-35 = young_adult, 35-60 = adult.

2. Step 2: Assign each age to bin

   15 -> child, 22 -> young_adult, 37 -> adult, 45 -> adult.

3. Final Answer:

   [child, young_adult, adult, adult] -> Option C

4. Quick Check:

   Age 37 and 45 fall in adult bin [OK]

Hint: Match values to bins to find labels [OK]

Common Mistakes:

• Misunderstanding bin edges (inclusive/exclusive)
• Assigning wrong labels to ages
• Confusing order of labels

4. What is wrong with this code to create a new column year from a datetime column date in DataFrame df?

df['year'] = df['date'].year

medium

A. There is no error; the code is correct

B. You must use dt.year to access year from datetime column

C. You should use df.year = df.date.year instead

D. The column date must be converted to string first

Solution

1. Step 1: Recall how to extract year from datetime column

   Use df['date'].dt.year to get year from pandas datetime column.

2. Step 2: Identify error in code

   Code uses df['date'].year which is invalid; .year is not attribute of Series.

3. Final Answer:

   You must use dt.year to access year from datetime column -> Option B

4. Quick Check:

   Use df['date'].dt.year for datetime year [OK]

Hint: Use .dt accessor for datetime column parts [OK]

Common Mistakes:

• Using .year directly on Series instead of .dt.year
• Converting datetime to string unnecessarily
• Assigning to df.year instead of df['year']

5. You have a DataFrame df with columns height_cm and weight_kg. You want to create a new feature bmi using the formula bmi = weight_kg / (height_m)^2, where height in meters is height_cm divided by 100. Which code correctly creates the bmi column?

hard

A. df['bmi'] = df['weight_kg'] / (df['height_cm'] / 100) ** 2

B. df['bmi'] = df['weight_kg'] / df['height_cm'] ** 2 / 10000

C. df['bmi'] = df['weight_kg'] * (df['height_cm'] / 100) ** 2

D. df['bmi'] = df['weight_kg'] / (df['height_cm'] * 100) ** 2

Solution

1. Step 1: Convert height from cm to meters

   Height in meters = height_cm / 100.

2. Step 2: Apply BMI formula correctly

   BMI = weight_kg / (height_m)^2 = weight_kg / (height_cm / 100)^2.

3. Step 3: Check each option

   df['bmi'] = df['weight_kg'] / (df['height_cm'] / 100) ** 2 matches formula exactly. Others have wrong operations or units.

4. Final Answer:

   df['bmi'] = df['weight_kg'] / (df['height_cm'] / 100) ** 2 -> Option A

5. Quick Check:

   BMI formula with height in meters = weight / height² [OK]

Hint: Divide height_cm by 100 before squaring for BMI [OK]

Common Mistakes:

• Forgetting to convert cm to meters
• Multiplying instead of dividing
• Incorrect order of operations