Missing data strategies decision in Pandas - Step-by-Step Execution

Concept Flow - Missing data strategies decision

Start with DataFrame

↓

Check for missing values

↓

Decide strategy

↓

Drop

↓

Resulting DataFrame

↓

End

Start with a DataFrame, check for missing values, decide to drop, fill, or leave them, then get the final DataFrame.

Execution Sample

Pandas

import pandas as pd

df = pd.DataFrame({'A': [1, None, 3], 'B': [4, 5, None]})

# Strategy: fill missing with 0
df_filled = df.fillna(0)
print(df_filled)

This code creates a DataFrame with missing values and fills them with zero.

Execution Table

Step	Action	DataFrame State	Missing Count	Result
1	Create DataFrame	{'A': [1, None, 3], 'B': [4, 5, None]}	2	Initial data with 2 missing values
2	Check missing	Same as step 1	2	Found 2 missing values
3	Fill missing with 0	{'A': [1, 0, 3], 'B': [4, 5, 0]}	0	Missing values replaced by 0
4	Print filled DataFrame	Same as step 3	0	Output shows no missing values
5	End	Final DataFrame	0	Process complete

💡 All missing values handled by filling with zero, no missing values remain.

Variable Tracker

Variable	Start	After fillna	Final
df	{'A': [1, None, 3], 'B': [4, 5, None]}	Same	Same
df_filled	N/A	{'A': [1, 0, 3], 'B': [4, 5, 0]}	Same

Key Moments - 2 Insights

Why do missing values still appear in 'df' after filling in 'df_filled'?

Because 'fillna' returns a new DataFrame and does not change 'df' unless assigned back. See execution_table step 3 where 'df_filled' is created separately.

What happens if we drop missing values instead of filling?

Rows with missing values are removed, reducing data size. This is a different strategy not shown here but would change DataFrame shape.

Visual Quiz - 3 Questions

Look at the execution table, how many missing values are in the DataFrame after step 3?

A 0

B 1

C 2

D 3

💡 Hint

Check the 'Missing Count' column at step 3 in the execution_table.

According to variable_tracker, what is the value of 'df_filled' after fillna?

A {'A': [1, None, 3], 'B': [4, 5, None]}

B {'A': [1, 0, 3], 'B': [4, 5, 0]}

C N/A

D Empty DataFrame

💡 Hint

Look at the 'After fillna' column for 'df_filled' in variable_tracker.

If we changed the strategy to drop missing values, what would happen to the DataFrame size?

A It would increase

B It would stay the same

C It would decrease

D It would become empty

💡 Hint

Dropping missing values removes rows with missing data, reducing size.

Concept Snapshot

Missing data strategies:
- Check missing values with isna()
- Decide: drop rows, fill values, or leave
- fillna() fills missing values
- dropna() removes missing rows
- Choose strategy based on data and goal

Full Transcript

We start with a DataFrame that has missing values. We check how many missing values exist. Then we decide how to handle them. One way is to fill missing values with a number like zero. This creates a new DataFrame with no missing values. The original DataFrame stays the same unless we overwrite it. Another way is to drop rows with missing values, which reduces the data size. Choosing the right strategy depends on the data and what you want to do next.

Practice

1.

What is the main purpose of using dropna() in pandas?

easy

A. To remove rows or columns that contain missing data

B. To replace missing data with zeros

C. To fill missing data with the mean value

D. To convert data types of columns

Solution

1. Step 1: Understand dropna() function

   dropna() removes rows or columns that have missing values (NaN).

2. Step 2: Compare with other options

   Replacing missing data or converting data types are done by other functions like fillna() or astype().

3. Final Answer:

   To remove rows or columns that contain missing data -> Option A

4. Quick Check:

   dropna() removes missing data rows/columns [OK]

Hint: Remember: dropna drops missing data, fillna fills it [OK]

Common Mistakes:

• Confusing dropna with fillna
• Thinking dropna fills missing values
• Assuming dropna changes data types

2.

Which of the following is the correct syntax to fill missing values in a DataFrame df with the value 0?

?

easy

A. df.fillna(0)

B. df.dropna(0)

C. df.fill(na=0)

D. df.replace(na=0)

Solution

1. Step 1: Recall fillna syntax

   The correct method to fill missing values is fillna() with the value as argument.

2. Step 2: Check each option

   df.fillna(0) uses df.fillna(0) which is correct. Others have wrong method names or parameters.

3. Final Answer:

   df.fillna(0) -> Option A

4. Quick Check:

   Use fillna(value) to fill missing values [OK]

Hint: fillna fills missing values; dropna removes rows/columns [OK]

Common Mistakes:

• Using dropna instead of fillna
• Wrong method name like fill() or replace()
• Passing parameters incorrectly

3.

Given the DataFrame df below, what will be the output of df.fillna(method='ffill')?

import pandas as pd
df = pd.DataFrame({'A': [1, None, 3, None], 'B': [None, 2, None, 4]})
print(df.fillna(method='ffill'))

medium

A. Removes rows with missing values

B. Fills missing values with zero

C. Fills missing values by propagating last valid observation forward

D. Raises an error due to wrong method

Solution

1. Step 1: Understand forward fill method

   method='ffill' fills missing values by copying the last known non-missing value forward.

2. Step 2: Apply to DataFrame

   For column A: 1 stays, None becomes 1, 3 stays, None becomes 3. For B: None stays None (no previous), 2 stays, None becomes 2, 4 stays.

3. Final Answer:

   Fills missing values by propagating last valid observation forward -> Option C

4. Quick Check:

   ffill copies last valid value forward [OK]

Hint: ffill copies previous value forward to fill NaNs [OK]

Common Mistakes:

• Thinking fillna(method='ffill') fills with zero
• Confusing ffill with dropping rows
• Expecting error from method parameter

4.

What is wrong with the following code if the goal is to remove all rows with missing data from DataFrame df?

df.fillna(drop=True)

medium

A. The code should use df.dropna(inplace=True) to modify in place

B. fillna() does not have a drop parameter; use dropna() instead

C. The parameter drop=True should be dropna=True

D. The code is correct and will remove rows with missing data

Solution

1. Step 1: Check method and parameters

   fillna() is for filling missing values and does not accept a drop parameter.

2. Step 2: Correct method for removing rows

   To remove rows with missing data, use dropna() instead.

3. Final Answer:

   fillna() does not have a drop parameter; use dropna() instead -> Option B

4. Quick Check:

   Use dropna() to remove missing data rows [OK]

Hint: Use dropna() to remove rows, fillna() to replace values [OK]

Common Mistakes:

• Using fillna() with invalid parameters
• Confusing fillna and dropna functions
• Expecting fillna to remove rows

5.

You have a DataFrame df with many missing values scattered randomly. You want to keep as much data as possible but avoid rows with more than 2 missing values. Which pandas method and parameter combination is best?

hard

A. Use df.fillna(method='bfill') to fill missing values backward

B. Use df.fillna(0) to replace all missing values with zero

C. Use df.dropna(axis=1) to drop columns with missing values

D. Use df.dropna(thresh=len(df.columns)-2) to drop rows with more than 2 missing values

Solution

1. Step 1: Understand the threshold parameter

   dropna(thresh=n) keeps rows with at least n non-missing values.

2. Step 2: Calculate threshold for max 2 missing values

   If total columns are len(df.columns), then rows must have at least len(df.columns) - 2 non-missing values to keep.

3. Step 3: Evaluate other options

   Filling with zero or backward fill changes data but does not drop rows. Dropping columns removes data unnecessarily.

4. Final Answer:

   Use df.dropna(thresh=len(df.columns)-2) to drop rows with more than 2 missing values -> Option D

5. Quick Check:

   dropna(thresh) keeps rows with enough non-NaNs [OK]

Hint: Use dropna(thresh) to keep rows with few missing values [OK]

Common Mistakes:

• Using fillna instead of dropping rows
• Dropping columns instead of rows
• Misunderstanding thresh parameter