Dropping missing values (dropna) in Data Analysis Python - Step-by-Step Execution

Concept Flow - Dropping missing values (dropna)

Start with DataFrame

↓

Check each row/column for missing values

↓

Decide axis: rows or columns

↓

Remove rows/columns with missing values

↓

Return cleaned DataFrame

↓

End

The process checks for missing values in rows or columns, removes those containing missing data, and returns a cleaned DataFrame.

Execution Sample

Data Analysis Python

import pandas as pd

df = pd.DataFrame({
  'A': [1, 2, None],
  'B': [4, None, 6]
})

clean_df = df.dropna()

This code creates a DataFrame with missing values and removes rows that contain any missing values.

Execution Table

Step	DataFrame State	Action	Resulting DataFrame
1	{'A': [1, 2, None], 'B': [4, None, 6]}	Initial DataFrame with missing values	Same as input
2	Check row 0: A=1, B=4	No missing values in row 0	Row 0 kept
3	Check row 1: A=2, B=None	Missing value in B at row 1	Row 1 dropped
4	Check row 2: A=None, B=6	Missing value in A at row 2	Row 2 dropped
5	Final DataFrame after dropna()	Rows with missing values removed	{'A': [1.0], 'B': [4.0]}

💡 All rows with any missing values are removed, leaving only complete rows.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
df	{'A': [1, 2, None], 'B': [4, None, 6]}	Same	Same	Same	Same
clean_df	Undefined	Undefined	Undefined	Undefined	{'A': [1.0], 'B': [4.0]}

Key Moments - 3 Insights

Why does dropna() remove entire rows instead of just the missing values?

dropna() removes rows (or columns) that contain any missing values to keep data complete, as shown in execution_table rows 3 and 4 where rows with None are dropped entirely.

What happens if no axis is specified in dropna()?

By default, dropna() removes rows with missing values (axis=0), as demonstrated in the example where rows with missing data are dropped.

Can dropna() remove columns instead of rows?

Yes, by setting axis=1, dropna() removes columns with missing values. This is not shown here but works similarly to row removal.

Visual Quiz - 3 Questions

Look at the execution table, what is the state of the DataFrame after step 3?

A Only row 0 remains

B Rows 0 and 1 remain

C Rows 1 and 2 remain

D All rows remain

💡 Hint

Refer to execution_table row 3 where row 1 is dropped due to missing value.

At which step does the DataFrame lose row 2?

A Step 2

B Step 4

C Step 3

D Step 5

💡 Hint

Check execution_table row 4 where row 2 is dropped for missing value in column A.

If dropna(axis=1) was used instead, what would happen to the DataFrame?

A Rows with missing values would be dropped

B Nothing would change

C Columns with missing values would be dropped

D All data would be removed

💡 Hint

dropna with axis=1 removes columns containing missing values, not rows.

Concept Snapshot

dropna() removes missing data from DataFrames.
By default, it drops rows with any missing values.
Use axis=1 to drop columns instead.
Returns a new DataFrame without missing data.
Helps clean data for analysis.

Full Transcript

This visual execution shows how dropna() works in pandas. We start with a DataFrame containing missing values. Step by step, each row is checked for missing data. Rows with any missing values are removed. The final DataFrame contains only complete rows. Variables track the original and cleaned DataFrames. Key moments clarify why entire rows are dropped and how axis affects behavior. The quiz tests understanding of these steps and outcomes.

Practice

1. What does the dropna() function do in a pandas DataFrame?

easy

A. Duplicates rows with missing values

B. Fills missing values with zeros

C. Sorts the DataFrame by index

D. Removes rows or columns with missing values

Solution

1. Step 1: Understand the purpose of dropna()

   The dropna() function is used to remove missing data (NaN values) from a DataFrame.

2. Step 2: Identify what is removed

   It removes rows or columns that contain missing values, depending on parameters.

3. Final Answer:

   Removes rows or columns with missing values -> Option D

4. Quick Check:

   dropna() removes missing data [OK]

Hint: Remember: dropna means drop missing data [OK]

Common Mistakes:

• Confusing dropna with fillna
• Thinking it sorts data
• Assuming it duplicates data

2. Which of the following is the correct syntax to drop rows with any missing values in a DataFrame df?

easy

A. df.dropna(inplace=False, axis=2)

B. df.dropna(axis=1)

C. df.dropna(axis=0)

D. df.dropna(axis='columns')

Solution

1. Step 1: Understand axis parameter in dropna()

   axis=0 means drop rows, axis=1 means drop columns.

2. Step 2: Identify correct axis for dropping rows

   To drop rows with missing values, use axis=0.

3. Final Answer:

   df.dropna(axis=0) -> Option C

4. Quick Check:

   axis=0 drops rows [OK]

Hint: axis=0 drops rows, axis=1 drops columns [OK]

Common Mistakes:

• Using axis=1 to drop rows
• Using invalid axis=2
• Confusing inplace parameter

3. Given the DataFrame df below, what will df.dropna() return?

import pandas as pd
data = {'A': [1, 2, None], 'B': [4, None, 6]}
df = pd.DataFrame(data)
print(df.dropna())

medium

A. Rows with index 0 only

B. Rows with index 0 and 1

C. Rows with index 1 and 2

D. Empty DataFrame

Solution

1. Step 1: Examine the DataFrame rows for missing values

   Row 0: A=1, B=4 (no missing)
   Row 1: A=2, B=NaN (missing in B)
   Row 2: A=NaN, B=6 (missing in A)

2. Step 2: Apply dropna() default behavior

   By default, dropna() drops rows with any missing values, so only row 0 remains.

3. Final Answer:

   Rows with index 0 only -> Option A

4. Quick Check:

   Only complete rows remain [OK]

Hint: dropna() keeps only rows without any NaN [OK]

Common Mistakes:

• Assuming it drops columns instead of rows
• Thinking it fills missing values
• Expecting multiple rows to remain

4. What is wrong with this code snippet if the goal is to drop columns with any missing values?

df.dropna(axis=2)

medium

A. axis=2 is invalid; axis should be 0 or 1

B. dropna cannot drop columns

C. Missing inplace=True to modify df

D. dropna only works on rows by default

Solution

1. Step 1: Check valid axis values for dropna

   Valid axis values are 0 (rows) and 1 (columns). axis=2 is invalid and causes an error.

2. Step 2: Understand error cause

   Using axis=2 will raise a ValueError because pandas does not support axis=2 for DataFrames.

3. Final Answer:

   axis=2 is invalid; axis should be 0 or 1 -> Option A

4. Quick Check:

   axis must be 0 or 1 [OK]

Hint: Only axis=0 or 1 allowed in dropna [OK]

Common Mistakes:

• Using axis=2 by mistake
• Forgetting axis parameter
• Confusing inplace with axis

5. You have a DataFrame df with some missing values. You want to drop only those rows where all columns are missing values. Which dropna() parameter should you use?

hard

A. dropna(axis=1)

B. dropna(how='all')

C. dropna(thresh=2)

D. dropna(how='any')

Solution

1. Step 1: Understand the how parameter in dropna()

   how='all' drops rows only if all values are missing.
   how='any' drops rows if any value is missing.

2. Step 2: Identify correct parameter for dropping rows with all missing

   Since the goal is to drop rows where all columns are missing, use how='all'.

3. Final Answer:

   dropna(how='all') -> Option B

4. Quick Check:

   how='all' drops rows with all NaN [OK]

Hint: Use how='all' to drop rows with all missing [OK]

Common Mistakes:

• Using how='any' which drops too many rows
• Confusing thresh with how
• Dropping columns instead of rows