Dropping missing values with dropna() in Pandas - Step-by-Step Execution

Concept Flow - Dropping missing values with dropna()

Start with DataFrame

↓

Call dropna()

↓

Check each row/column for missing values

↓

Remove rows/columns with missing values

↓

Return cleaned DataFrame

↓

End

The flow starts with a DataFrame, then dropna() checks for missing values and removes rows or columns containing them, returning a cleaned DataFrame.

Execution Sample

Pandas

import pandas as pd

data = {'A': [1, 2, None], 'B': [4, None, 6]}
df = pd.DataFrame(data)
df_clean = df.dropna()

This code creates a DataFrame with missing values and uses dropna() to remove rows that have any missing values.

Execution Table

Step	DataFrame State	Action	Resulting DataFrame
1	{'A': [1, 2, None], 'B': [4, None, 6]}	Create DataFrame with missing values	A B 0 1.0 4.0 1 2.0 NaN 2 NaN 6.0
2	DataFrame with missing values	Call df.dropna() to remove rows with any NaN	A B 0 1.0 4.0
3	After dropna()	Return DataFrame with only complete rows	A B 0 1.0 4.0
4	No more rows with NaN	Stop execution	Final cleaned DataFrame

💡 All rows with any missing values are removed; only complete rows remain.

Variable Tracker

Variable	Start	After dropna()	Final
df	{'A': [1, 2, None], 'B': [4, None, 6]}	Same (unchanged)	Same (unchanged)
df_clean	Not defined	DataFrame with only complete rows	DataFrame with only complete rows

Key Moments - 3 Insights

Why does dropna() remove entire rows instead of just the missing values?

dropna() removes rows (or columns) that contain any missing values to keep data consistent. This is shown in execution_table step 2 where rows with NaN are removed entirely.

Does dropna() change the original DataFrame df?

No, dropna() returns a new DataFrame by default and does not modify df itself, as seen in variable_tracker where df remains unchanged after dropna().

What happens if all rows have missing values?

dropna() will return an empty DataFrame with no rows, because it removes all rows containing any NaN values, as implied by the exit condition in execution_table.

Visual Quiz - 3 Questions

Look at the execution table at step 2, how many rows remain after dropna()?

A 1 row

B 2 rows

C 3 rows

D 0 rows

💡 Hint

Check the 'Resulting DataFrame' column at step 2 in the execution_table.

According to variable_tracker, what is the value of df after dropna() is called?

A It is modified to remove missing values

B It remains unchanged

C It becomes empty

D It contains only missing values

💡 Hint

Look at the 'df' row in variable_tracker after dropna() is called.

If we want to remove columns with missing values instead of rows, which parameter should we change in dropna()?

A how='all'

B thresh=2

C axis=1

D subset=['A']

💡 Hint

Recall dropna() default removes rows; changing axis to 1 removes columns with NaN.

Concept Snapshot

pandas dropna() removes missing values by dropping rows or columns.
Syntax: df.dropna(axis=0) drops rows; axis=1 drops columns.
Returns a new DataFrame by default, original unchanged.
Use to clean data by removing incomplete entries.
Can customize with parameters like how, thresh, subset.

Full Transcript

We start with a DataFrame that has missing values. When we call dropna(), it checks each row for any missing values. If a row has any missing value, dropna() removes that entire row. The result is a new DataFrame with only complete rows. The original DataFrame stays the same. This process helps clean data by removing incomplete rows. If you want to remove columns instead, you can set axis=1. The execution table shows the DataFrame before and after dropna(), and the variable tracker confirms which variables change. Key points include understanding that dropna() removes whole rows or columns, not just the missing cells, and that it returns a new DataFrame without changing the original.

Practice

1. What does the dropna() function do in pandas?

easy

A. Sorts the DataFrame by index

B. Fills missing values with zeros

C. Removes rows or columns with missing values

D. Calculates the mean of columns

Solution

1. Step 1: Understand the purpose of dropna()

   The dropna() function is designed to remove data that contains missing values (NaN) from a DataFrame.

2. Step 2: Identify what dropna() removes

   It removes either rows or columns that have missing values, depending on the parameters used.

3. Final Answer:

   Removes rows or columns with missing values -> Option C

4. Quick Check:

   dropna() removes missing data [OK]

Hint: Remember: dropna() drops missing data, not fills it [OK]

Common Mistakes:

• Confusing dropna() with fillna()
• Thinking dropna() sorts data
• Assuming dropna() calculates statistics

2. Which of the following is the correct syntax to drop rows with any missing values in a DataFrame df?

easy

A. df.dropna(axis=0, how='any')

B. df.dropna(axis=1, how='any')

C. df.dropna(axis=0, how='all')

D. df.dropna(axis=1, how='all')

Solution

1. Step 1: Identify axis for rows

   In pandas, axis=0 means rows, so to drop rows, use axis=0.

2. Step 2: Use how='any' to drop rows with any missing values

   The how='any' parameter drops rows if any value is missing in that row.

3. Final Answer:

   df.dropna(axis=0, how='any') -> Option A

4. Quick Check:

   Drop rows with any NaN = axis=0, how='any' [OK]

Hint: axis=0 drops rows; how='any' drops if any NaN [OK]

Common Mistakes:

• Using axis=1 to drop rows
• Using how='all' to drop rows with any NaN
• Confusing axis and how parameters

3. Given the DataFrame:

import pandas as pd
data = {'A': [1, 2, None], 'B': [4, None, 6], 'C': [7, 8, 9]}
df = pd.DataFrame(data)

What is the result of df.dropna(axis=0, how='any')?

medium

A. Rows with index 0 and 1 only

B. Rows with index 0 and 2 only

C. All rows dropped

D. Only row with index 0

Solution

1. Step 1: Identify rows with any missing values

   Row 0: A=1, B=4, C=7 (no NaN)
   Row 1: A=2, B=NaN, C=8 (has NaN)
   Row 2: A=NaN, B=6, C=9 (has NaN)

2. Step 2: Drop rows with any NaN

   Only row 0 has no missing values, so only it remains after dropna(axis=0, how='any').

3. Final Answer:

   Only row with index 0 -> Option D

4. Quick Check:

   Drop rows with any NaN leaves only complete rows [OK]

Hint: Check which rows have any NaN to keep them [OK]

Common Mistakes:

• Keeping rows with NaN
• Confusing axis=0 and axis=1
• Misreading how='any' vs how='all'

4. You run the code:

df.dropna(axis=2)

and get an error. What is the problem?

medium

A. axis=2 is invalid; only 0 (rows) or 1 (columns) allowed

B. dropna() cannot be used on DataFrames

C. Missing values must be filled before dropping

D. You must specify how='any' or how='all'

Solution

1. Step 1: Understand valid axis values for dropna()

   In pandas, axis can only be 0 (rows) or 1 (columns) for dropna().

2. Step 2: Identify the error cause

   Using axis=2 is invalid and causes an error because it is not a supported axis.

3. Final Answer:

   axis=2 is invalid; only 0 (rows) or 1 (columns) allowed -> Option A

4. Quick Check:

   Valid axis values = 0 or 1 only [OK]

Hint: axis must be 0 or 1 for dropna() [OK]

Common Mistakes:

• Using axis=2 mistakenly
• Thinking dropna() needs fillna() first
• Assuming how parameter is mandatory

5. You have a DataFrame df with some rows having all values missing and some rows having only some missing values. Which dropna() call will remove only rows where all values are missing, keeping rows with partial data?

hard

A. df.dropna(axis=0, how='any')

B. df.dropna(axis=0, how='all')

C. df.dropna(axis=1, how='any')

D. df.dropna(axis=1, how='all')

Solution

1. Step 1: Understand how='all' parameter

   Using how='all' drops rows only if all values in that row are missing.

2. Step 2: Use axis=0 to drop rows

   Since we want to drop rows, axis=0 is correct.

3. Step 3: Confirm this keeps rows with some data

   Rows with some non-missing values are kept because not all values are missing.

4. Final Answer:

   df.dropna(axis=0, how='all') -> Option B

5. Quick Check:

   Drop rows with all NaN = axis=0, how='all' [OK]

Hint: Use how='all' to drop only fully empty rows [OK]

Common Mistakes:

• Using how='any' which drops partial rows
• Using axis=1 which drops columns
• Confusing how parameter effects