Feature engineering basics in Pandas - Step-by-Step Execution

Concept Flow - Feature engineering basics

Start with raw data

↓

Identify useful info

↓

Create new features

↓

Add features to data

↓

Use features for analysis or model

Feature engineering means making new useful columns from raw data to help analysis or models.

Execution Sample

Pandas

import pandas as pd

data = pd.DataFrame({'age': [25, 32, 47], 'income': [50000, 60000, 80000]})
data['age_group'] = pd.cut(data['age'], bins=[0,30,50], labels=['Young','Old'])

This code creates a new column 'age_group' by grouping ages into 'Young' or 'Old'.

Execution Table

Step	Action	DataFrame State	New Feature Created
1	Create DataFrame with 'age' and 'income'	{'age': [25,32,47], 'income': [50000,60000,80000]}	None
2	Apply pd.cut to 'age' with bins [0,30,50]	Same as step 1	Series with ['Young', 'Young', 'Old']
3	Assign new series to 'age_group' column	{'age': [25,32,47], 'income': [50000,60000,80000], 'age_group': ['Young','Young','Old']}	'age_group' added
4	Final DataFrame ready for use	{'age': [25,32,47], 'income': [50000,60000,80000], 'age_group': ['Young','Young','Old']}	'age_group' present

💡 New feature 'age_group' created and added to DataFrame for better analysis.

Variable Tracker

Variable	Start	After Step 2	After Step 3	Final
data	Empty	{'age': [25,32,47], 'income': [50000,60000,80000]}	{'age': [25,32,47], 'income': [50000,60000,80000], 'age_group': ['Young','Young','Old']}	{'age': [25,32,47], 'income': [50000,60000,80000], 'age_group': ['Young','Young','Old']}
age_group	None	Series(['Young','Young','Old'])	Assigned to data['age_group']	Present in data

Key Moments - 2 Insights

Why do we use pd.cut to create 'age_group' instead of just copying 'age'?

pd.cut groups continuous numbers into categories, making it easier to analyze age ranges. See execution_table step 2 where pd.cut creates categories.

Does adding a new column change the original data or create a copy?

Adding a new column modifies the original DataFrame in place, as shown in execution_table step 3 where 'age_group' is added.

Visual Quiz - 3 Questions

Look at the execution table at step 2, what does pd.cut produce?

A A new DataFrame

B A Series with categorical labels

C A list of numbers

D An error

💡 Hint

Check the 'New Feature Created' column at step 2 in execution_table.

At which step is the new feature 'age_group' added to the DataFrame?

A Step 3

B Step 2

C Step 1

D Step 4

💡 Hint

Look at the 'Action' and 'DataFrame State' columns in execution_table.

If we changed bins in pd.cut to [0, 20, 40, 60], how would the 'age_group' Series change?

A It would become numeric

B It would stay the same

C It would have three categories instead of two

D It would cause an error

💡 Hint

Think about how pd.cut uses bins to create categories, see variable_tracker for 'age_group'.

Concept Snapshot

Feature engineering means creating new columns from raw data.
Use pandas functions like pd.cut to group or transform data.
Add new features to your DataFrame for better analysis or models.
New features can be categories, numbers, or text.
Always check your new feature values before use.

Full Transcript

Feature engineering basics means making new columns from your data to help analysis or machine learning. For example, using pandas, you can create a new column 'age_group' by grouping ages into categories like 'Young' or 'Old' using pd.cut. The process starts with raw data, then you identify useful info, create new features, add them to your data, and finally use them for analysis or models. In the example, we start with a DataFrame with 'age' and 'income'. We use pd.cut to create a series of age groups, then add this as a new column. This helps simplify age data into categories. Adding a new column changes the original DataFrame. Understanding how new features are created and added is key to feature engineering.

Practice

1. What is the main purpose of feature engineering in pandas?

easy

A. To create new useful columns from existing data

B. To delete unnecessary rows from the dataset

C. To change the data type of columns

D. To visualize data using charts

Solution

1. Step 1: Understand feature engineering definition

   Feature engineering means creating new columns that help models learn better.

2. Step 2: Compare options with definition

   Only To create new useful columns from existing data matches the purpose of creating new useful columns from existing data.

3. Final Answer:

   To create new useful columns from existing data -> Option A

4. Quick Check:

   Feature engineering = create new columns [OK]

Hint: Feature engineering = making new columns from old data [OK]

Common Mistakes:

• Confusing feature engineering with data cleaning
• Thinking it only changes data types
• Mixing it up with data visualization

2. Which of the following is the correct way to create a new column total as sum of columns 2a and 2b in a pandas DataFrame df?

easy

A. df['total'] <- df['2a'] + df['2b']

B. df.total = df.2a + df.2b

C. df['total'] == df['2a'] + df['2b']

D. df['total'] = df['2a'] + df['2b']

Solution

1. Step 1: Recall pandas syntax for new column

   Use df['new_col'] = expression to create new columns.

2. Step 2: Check each option

   df['total'] = df['2a'] + df['2b'] uses correct syntax. df.total = df.2a + df.2b misses brackets, A uses wrong operator, D uses comparison operator.

3. Final Answer:

   df['total'] = df['2a'] + df['2b'] -> Option D

4. Quick Check:

   New column = df['col'] = ... [OK]

Hint: Use df['col'] = ... to add new columns [OK]

Common Mistakes:

• Using == instead of = for assignment
• Using <- which is not Python syntax
• Accessing columns without brackets

3. Given the DataFrame df below, what will be the output of df['age_group'] after this code?

import pandas as pd
df = pd.DataFrame({'age': [15, 22, 37, 45]})
df['age_group'] = pd.cut(df['age'], bins=[0, 18, 35, 60], labels=['child', 'young_adult', 'adult'])

medium

A. [child, young_adult, young_adult, adult]

B. [child, child, young_adult, adult]

C. [child, young_adult, adult, adult]

D. [child, young_adult, adult, young_adult]

Solution

1. Step 1: Understand pd.cut bins and labels

   Bins: 0-18 = child, 18-35 = young_adult, 35-60 = adult.

2. Step 2: Assign each age to bin

   15 -> child, 22 -> young_adult, 37 -> adult, 45 -> adult.

3. Final Answer:

   [child, young_adult, adult, adult] -> Option C

4. Quick Check:

   Age 37 and 45 fall in adult bin [OK]

Hint: Match values to bins to find labels [OK]

Common Mistakes:

• Misunderstanding bin edges (inclusive/exclusive)
• Assigning wrong labels to ages
• Confusing order of labels

4. What is wrong with this code to create a new column year from a datetime column date in DataFrame df?

df['year'] = df['date'].year

medium

A. There is no error; the code is correct

B. You must use dt.year to access year from datetime column

C. You should use df.year = df.date.year instead

D. The column date must be converted to string first

Solution

1. Step 1: Recall how to extract year from datetime column

   Use df['date'].dt.year to get year from pandas datetime column.

2. Step 2: Identify error in code

   Code uses df['date'].year which is invalid; .year is not attribute of Series.

3. Final Answer:

   You must use dt.year to access year from datetime column -> Option B

4. Quick Check:

   Use df['date'].dt.year for datetime year [OK]

Hint: Use .dt accessor for datetime column parts [OK]

Common Mistakes:

• Using .year directly on Series instead of .dt.year
• Converting datetime to string unnecessarily
• Assigning to df.year instead of df['year']

5. You have a DataFrame df with columns height_cm and weight_kg. You want to create a new feature bmi using the formula bmi = weight_kg / (height_m)^2, where height in meters is height_cm divided by 100. Which code correctly creates the bmi column?

hard

A. df['bmi'] = df['weight_kg'] / (df['height_cm'] / 100) ** 2

B. df['bmi'] = df['weight_kg'] / df['height_cm'] ** 2 / 10000

C. df['bmi'] = df['weight_kg'] * (df['height_cm'] / 100) ** 2

D. df['bmi'] = df['weight_kg'] / (df['height_cm'] * 100) ** 2

Solution

1. Step 1: Convert height from cm to meters

   Height in meters = height_cm / 100.

2. Step 2: Apply BMI formula correctly

   BMI = weight_kg / (height_m)^2 = weight_kg / (height_cm / 100)^2.

3. Step 3: Check each option

   df['bmi'] = df['weight_kg'] / (df['height_cm'] / 100) ** 2 matches formula exactly. Others have wrong operations or units.

4. Final Answer:

   df['bmi'] = df['weight_kg'] / (df['height_cm'] / 100) ** 2 -> Option A

5. Quick Check:

   BMI formula with height in meters = weight / height² [OK]

Hint: Divide height_cm by 100 before squaring for BMI [OK]

Common Mistakes:

• Forgetting to convert cm to meters
• Multiplying instead of dividing
• Incorrect order of operations